[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:41:46 dist-test-slave-gp6n krb5kdc[24081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:41:46 dist-test-slave-gp6n krb5kdc[24081](info): set up 2 sockets
Aug 08 15:41:46 dist-test-slave-gp6n krb5kdc[24081](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 15:41:47.977265 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:41:48 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:41:48.178753 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37797
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:48.519876 24097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:48.520160 24097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:48.520370 24097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:48.527633 24097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:41:48.527745 24097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:48.527853 24097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:48.527956 24097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:41:48.528043 24097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:41:48.538302 24097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:48.539696 24097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:48.541093 24097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:48.553583 24103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:48.553545 24106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:48.553622 24102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:48.554311 24097 server_base.cc:1034] running on GCE node
I20260808 15:41:48.555199 24097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:48.556568 24097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:48.557816 24097 hybrid_clock.cc:648] HybridClock initialized: now 1786203708557791 us; error 88 us; skew 500 ppm
I20260808 15:41:48.558185 24097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:48 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:48.561510 24097 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:41:48.562978 24097 webserver.cc:460] Webserver started at http://127.23.130.62:35315/ using document root <none> and password file <none>
I20260808 15:41:48.563670 24097 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:48.563817 24097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:48.564195 24097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:48.567327 24097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "c557aa2ff81c4dcb99f314ad1795e8b6"
format_stamp: "Formatted at 2026-08-08 15:41:48 on dist-test-slave-gp6n"
server_key: "5c874db83c849a1636500884f009be99"
server_key_iv: "71ac570811675642e3bf2ff69363ac5e"
server_key_version: "encryptionkey@0"
I20260808 15:41:48.568132 24097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "c557aa2ff81c4dcb99f314ad1795e8b6"
format_stamp: "Formatted at 2026-08-08 15:41:48 on dist-test-slave-gp6n"
server_key: "5c874db83c849a1636500884f009be99"
server_key_iv: "71ac570811675642e3bf2ff69363ac5e"
server_key_version: "encryptionkey@0"
I20260808 15:41:48.573509 24097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:41:48.577265 24114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:48.578735 24097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:41:48.578945 24097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "c557aa2ff81c4dcb99f314ad1795e8b6"
format_stamp: "Formatted at 2026-08-08 15:41:48 on dist-test-slave-gp6n"
server_key: "5c874db83c849a1636500884f009be99"
server_key_iv: "71ac570811675642e3bf2ff69363ac5e"
server_key_version: "encryptionkey@0"
I20260808 15:41:48.579169 24097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:48.603642 24097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:48.608515 24097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:48.608834 24097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:48.629582 24097 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37797
I20260808 15:41:48.629606 24170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37797 every 8 connection(s)
I20260808 15:41:48.631173 24097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:41:48.635151 24171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:48.637733 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24097
I20260808 15:41:48.638083 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:41:48.640568 24072 external_mini_cluster.cc:1428] Setting key 76ad679216aeb03c1c7a22aeda2394b3
I20260808 15:41:48.648384 24171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6: Bootstrap starting.
I20260808 15:41:48.651355 24171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:48.652375 24171 log.cc:824] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:48.655359 24171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6: No bootstrap required, opened a new log
I20260808 15:41:48.664790 24171 raft_consensus.cc:348] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } }
I20260808 15:41:48.665302 24171 raft_consensus.cc:374] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:48.665434 24171 raft_consensus.cc:729] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c557aa2ff81c4dcb99f314ad1795e8b6, State: Initialized, Role: FOLLOWER
I20260808 15:41:48.666163 24171 consensus_queue.cc:260] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } }
I20260808 15:41:48.666520 24171 raft_consensus.cc:388] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:41:48.666684 24171 raft_consensus.cc:482] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:41:48.666857 24171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:48.669031 24171 raft_consensus.cc:504] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } }
I20260808 15:41:48.669570 24171 leader_election.cc:302] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c557aa2ff81c4dcb99f314ad1795e8b6; no voters:
I20260808 15:41:48.670168 24171 leader_election.cc:288] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:41:48.670336 24177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 1 FOLLOWER]: Leader election won for term 1
Aug 08 15:41:48 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:48.671754 24177 raft_consensus.cc:686] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [term 1 LEADER]: Becoming Leader. State: Replica: c557aa2ff81c4dcb99f314ad1795e8b6, State: Running, Role: LEADER
I20260808 15:41:48.672569 24171 sys_catalog.cc:564] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:41:48.672425 24177 consensus_queue.cc:237] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } }
I20260808 15:41:48.676203 24178 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } } }
I20260808 15:41:48.676630 24178 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:48.681457 24179 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c557aa2ff81c4dcb99f314ad1795e8b6. Latest consensus state: current_term: 1 leader_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557aa2ff81c4dcb99f314ad1795e8b6" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37797 } } }
I20260808 15:41:48.682318 24179 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:48.689774 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:48.645291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.23.130.62:37797)
0808 15:41:48.646292 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:48.646312 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:48.647614 (+ 1302us) server_negotiation.cc:407] Connection header received
0808 15:41:48.657215 (+ 9601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:48.657242 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:48.657724 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:48.658167 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:48.661015 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:48.663151 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:48.664849 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:48.665981 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:48.666161 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:48.673515 (+ 7354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:48.673528 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:48.673550 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:48.673598 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:48.679311 (+ 5713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:48.680714 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:48.680726 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:48.680736 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:48.681025 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:48.681736 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:48.681743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:48.681749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:48.682829 (+ 1080us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:48.686675 (+ 3846us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:48.686983 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 15:41:48.687434 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":433,"threads_started":1}
I20260808 15:41:48.693179 24186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:41:48.697516 24186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:41:48.705858 24186 catalog_manager.cc:1347] Generated new cluster ID: d0b2621ed6074e5ca3f677b58791f103
I20260808 15:41:48.705993 24186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:41:48.725761 24186 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:41:48.727223 24186 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:41:48.740620 24186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c557aa2ff81c4dcb99f314ad1795e8b6: Generated new TSK 0
I20260808 15:41:48.741596 24186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:41:48.803107 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:49.150455 24200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:49.150753 24200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:49.150914 24200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:49.158545 24200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:49.158659 24200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:49.158834 24200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:41:49.168972 24200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:49.170676 24200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:49.172011 24200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:49.185911 24205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:49.185936 24208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:49.186085 24206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:49.186650 24200 server_base.cc:1034] running on GCE node
I20260808 15:41:49.187342 24200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:49.188367 24200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:49.189569 24200 hybrid_clock.cc:648] HybridClock initialized: now 1786203709189511 us; error 53 us; skew 500 ppm
I20260808 15:41:49.189886 24200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:49 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203709, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:49.193197 24200 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:41:49.194607 24200 webserver.cc:460] Webserver started at http://127.23.130.1:42499/ using document root <none> and password file <none>
I20260808 15:41:49.195225 24200 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:49.195336 24200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:49.195600 24200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:49.198577 24200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "94c69c93bc034675b6f8b4ae17a2ff23"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "0bbdca12b1ee52cee8282ce105fa895d"
server_key_iv: "5cb0289f5b833a04cd7ca6098b121652"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.199343 24200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "94c69c93bc034675b6f8b4ae17a2ff23"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "0bbdca12b1ee52cee8282ce105fa895d"
server_key_iv: "5cb0289f5b833a04cd7ca6098b121652"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.204788 24200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:41:49.208318 24216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.209717 24200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:41:49.209895 24200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "94c69c93bc034675b6f8b4ae17a2ff23"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "0bbdca12b1ee52cee8282ce105fa895d"
server_key_iv: "5cb0289f5b833a04cd7ca6098b121652"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.210101 24200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:49.244957 24200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:49.249220 24200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:49.249542 24200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:49.250658 24200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:49.252018 24200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:49.252125 24200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.252279 24200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:49.252362 24200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.290282 24200 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:38305
I20260808 15:41:49.290357 24343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:38305 every 8 connection(s)
I20260808 15:41:49.291754 24200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:41:49.293776 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24200
I20260808 15:41:49.294023 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:41:49.294782 24072 external_mini_cluster.cc:1428] Setting key 2197e0389bc478e4c20206cb2fd0a377
Aug 08 15:41:49 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203709, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
I20260808 15:41:49.314249 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:49.295007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:33187 (local address 127.23.130.62:37797)
0808 15:41:49.295576 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:49.295585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:49.298480 (+ 2895us) server_negotiation.cc:407] Connection header received
0808 15:41:49.299041 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:49.299047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:49.299139 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:49.299276 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:49.300885 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.301708 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.303517 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.303983 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.304137 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:49.307922 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:49.307928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:49.307932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:49.307967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:49.310978 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:49.311773 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:49.311781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:49.311790 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:49.311917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:49.312430 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:49.312434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:49.312436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:49.312708 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:49.312831 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:49.313735 (+ 904us) server_negotiation.cc:299] Negotiation successful
0808 15:41:49.313924 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
I20260808 15:41:49.315495 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:49.295737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37797 (local address 127.23.130.1:33187)
0808 15:41:49.296682 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:49.296740 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:49.298757 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:49.299535 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:49.299543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:49.300100 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:49.300697 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:49.300715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.301886 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.301892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:49.303348 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:49.303360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.304235 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.304245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:49.304511 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:49.305260 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:49.305290 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:49.307666 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:49.311133 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:49.311143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:49.311161 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:49.311634 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:49.312038 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:49.312044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:49.312048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:49.312253 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:49.312845 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:49.312851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:49.313228 (+ 377us) client_negotiation.cc:769] Sending connection context
0808 15:41:49.313580 (+ 352us) client_negotiation.cc:240] Negotiation successful
0808 15:41:49.314061 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":405,"threads_started":1}
I20260808 15:41:49.317128 24344 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37797
I20260808 15:41:49.317454 24344 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:49.318198 24344 heartbeater.cc:511] Master 127.23.130.62:37797 requested a full tablet report, sending...
I20260808 15:41:49.320418 24135 ts_manager.cc:194] Registered new tserver with Master: 94c69c93bc034675b6f8b4ae17a2ff23 (127.23.130.1:38305)
I20260808 15:41:49.321957 24135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:33187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:41:49.339787 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:49.676506 24350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:49.676843 24350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:49.677029 24350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:49.685106 24350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:49.685263 24350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:49.685441 24350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:41:49.695523 24350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:49.696875 24350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:49.698295 24350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:49.711755 24357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:49.712008 24358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:49.712283 24360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:49.713300 24350 server_base.cc:1034] running on GCE node
I20260808 15:41:49.713943 24350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:49.715091 24350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:49.716316 24350 hybrid_clock.cc:648] HybridClock initialized: now 1786203709716268 us; error 78 us; skew 500 ppm
I20260808 15:41:49.716639 24350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:49 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203709, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:49.719885 24350 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:41:49.721395 24350 webserver.cc:460] Webserver started at http://127.23.130.2:38069/ using document root <none> and password file <none>
I20260808 15:41:49.722084 24350 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:49.722232 24350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:49.722616 24350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:49.725471 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "cc8dfca241c84569a9acd87fbcdc24c2"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "b8f6702f8a2977df2e06a8a9dcf0b981"
server_key_iv: "cdaa52c7c7c5f8a9603ed3d03d289199"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.726248 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "cc8dfca241c84569a9acd87fbcdc24c2"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "b8f6702f8a2977df2e06a8a9dcf0b981"
server_key_iv: "cdaa52c7c7c5f8a9603ed3d03d289199"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.731387 24350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:41:49.734939 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.736244 24350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:41:49.736426 24350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "cc8dfca241c84569a9acd87fbcdc24c2"
format_stamp: "Formatted at 2026-08-08 15:41:49 on dist-test-slave-gp6n"
server_key: "b8f6702f8a2977df2e06a8a9dcf0b981"
server_key_iv: "cdaa52c7c7c5f8a9603ed3d03d289199"
server_key_version: "encryptionkey@0"
I20260808 15:41:49.736627 24350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:49.756290 24350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:49.760824 24350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:49.761132 24350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:49.762178 24350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:49.763609 24350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:49.763716 24350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.763871 24350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:49.763947 24350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:49.800990 24350 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:40347
I20260808 15:41:49.801046 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:40347 every 8 connection(s)
I20260808 15:41:49.802296 24350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:41:49.808436 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24350
I20260808 15:41:49.808645 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:41:49.809221 24072 external_mini_cluster.cc:1428] Setting key 92dc5a05a0035df5042c8283f6da93ab
Aug 08 15:41:49 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203709, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:49.824597 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:49.805059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:40431 (local address 127.23.130.62:37797)
0808 15:41:49.805279 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:49.805286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:49.806555 (+ 1269us) server_negotiation.cc:407] Connection header received
0808 15:41:49.808484 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:49.808488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:49.808593 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:49.808695 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:49.810736 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.811612 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.813153 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.813712 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.813852 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:49.817497 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:49.817506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:49.817513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:49.817555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:49.821304 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:49.822025 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:49.822032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:49.822038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:49.822199 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:49.822667 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:49.822675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:49.822680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:49.823048 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:49.823225 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:49.823868 (+ 643us) server_negotiation.cc:299] Negotiation successful
0808 15:41:49.824135 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:41:49.825215 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:49.805564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37797 (local address 127.23.130.2:40431)
0808 15:41:49.806334 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:49.806382 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:49.808174 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:49.808976 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:49.808987 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:49.809763 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:49.810568 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:49.810589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.811769 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.811779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:49.813000 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:49.813011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:49.814005 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:49.814015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:49.814219 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:49.815019 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:49.815056 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:49.817252 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:49.821443 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:49.821449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:49.821462 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:49.821888 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:49.822310 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:49.822313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:49.822316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:49.822552 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:49.823166 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:49.823171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:49.823470 (+ 299us) client_negotiation.cc:769] Sending connection context
0808 15:41:49.823687 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 15:41:49.824065 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":280,"threads_started":1}
I20260808 15:41:49.826814 24496 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37797
I20260808 15:41:49.827107 24496 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:49.827819 24496 heartbeater.cc:511] Master 127.23.130.62:37797 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:41:49.829330 24135 ts_manager.cc:194] Registered new tserver with Master: cc8dfca241c84569a9acd87fbcdc24c2 (127.23.130.2:40347)
I20260808 15:41:49.830250 24135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:40431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:41:49.856002 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:50.197741 24501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:50.198031 24501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:50.198133 24501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:50.205430 24501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:50.205542 24501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:50.205658 24501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:41:50.215585 24501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:50.217250 24501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:50.218725 24501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:50.234297 24510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:50.234360 24508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:50.234635 24507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:50.240051 24501 server_base.cc:1034] running on GCE node
I20260808 15:41:50.240880 24501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:50.241977 24501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:50.243232 24501 hybrid_clock.cc:648] HybridClock initialized: now 1786203710243172 us; error 56 us; skew 500 ppm
I20260808 15:41:50.243608 24501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:50.246912 24501 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:41:50.248752 24501 webserver.cc:460] Webserver started at http://127.23.130.3:40031/ using document root <none> and password file <none>
I20260808 15:41:50.249436 24501 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:50.249537 24501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:50.249794 24501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:50.252715 24501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "ef4ca7d240dd42c28a149e118136fcf3"
format_stamp: "Formatted at 2026-08-08 15:41:50 on dist-test-slave-gp6n"
server_key: "a6b1cec99ec7b696547948fbc0f68d44"
server_key_iv: "69a4886bd967e1145169cd839bfeffd2"
server_key_version: "encryptionkey@0"
I20260808 15:41:50.253520 24501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "ef4ca7d240dd42c28a149e118136fcf3"
format_stamp: "Formatted at 2026-08-08 15:41:50 on dist-test-slave-gp6n"
server_key: "a6b1cec99ec7b696547948fbc0f68d44"
server_key_iv: "69a4886bd967e1145169cd839bfeffd2"
server_key_version: "encryptionkey@0"
I20260808 15:41:50.258759 24501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:41:50.262156 24517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:50.263553 24501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:41:50.263725 24501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "ef4ca7d240dd42c28a149e118136fcf3"
format_stamp: "Formatted at 2026-08-08 15:41:50 on dist-test-slave-gp6n"
server_key: "a6b1cec99ec7b696547948fbc0f68d44"
server_key_iv: "69a4886bd967e1145169cd839bfeffd2"
server_key_version: "encryptionkey@0"
I20260808 15:41:50.263942 24501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:50.304121 24501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:50.308338 24501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:50.308662 24501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:50.309710 24501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:50.311164 24501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:50.311246 24501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:50.311414 24501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:50.311475 24501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:50.325016 24344 heartbeater.cc:503] Master 127.23.130.62:37797 was elected leader, sending a full tablet report...
I20260808 15:41:50.349227 24501 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:37797
I20260808 15:41:50.349304 24644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:37797 every 8 connection(s)
I20260808 15:41:50.350663 24501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:41:50.355050 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24501
I20260808 15:41:50.355258 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:41:50.355836 24072 external_mini_cluster.cc:1428] Setting key 8c9be4e3b4ed9cbc7e5362d1eadca76e
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:50.371884 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.353357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:39873 (local address 127.23.130.62:37797)
0808 15:41:50.353863 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.353868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.355073 (+ 1205us) server_negotiation.cc:407] Connection header received
0808 15:41:50.357652 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.357657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.357744 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.357876 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.359616 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.360420 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.361966 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.362477 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.362604 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.365985 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.365990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.365993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.366025 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.368861 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.369527 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.369531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.369533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.369648 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.370036 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.370041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.370044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.370314 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.370524 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.371241 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.371483 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260808 15:41:50.372715 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.353984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37797 (local address 127.23.130.3:39873)
0808 15:41:50.354880 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:50.354933 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:50.357367 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:50.358112 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:50.358121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:50.358790 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:50.359455 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:50.359474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.360569 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.360576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:50.361813 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:50.361827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.362700 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.362707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:50.362933 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.363527 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:50.363548 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:50.365784 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:50.368973 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:50.368979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:50.368989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:50.369423 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:50.369741 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:50.369744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:50.369746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:50.369922 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:50.370486 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:50.370495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:50.370881 (+ 386us) client_negotiation.cc:769] Sending connection context
0808 15:41:50.371105 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 15:41:50.371564 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":286,"threads_started":1}
I20260808 15:41:50.374094 24646 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37797
I20260808 15:41:50.374434 24646 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:50.375181 24646 heartbeater.cc:511] Master 127.23.130.62:37797 requested a full tablet report, sending...
I20260808 15:41:50.376648 24135 ts_manager.cc:194] Registered new tserver with Master: ef4ca7d240dd42c28a149e118136fcf3 (127.23.130.3:37797)
I20260808 15:41:50.377533 24135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:39873
I20260808 15:41:50.384188 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:41:50.434649 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.418160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.23.130.1:38305)
0808 15:41:50.418902 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.418911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.418941 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:41:50.419117 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.419124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.419359 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.419567 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.421261 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.422293 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.423699 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.424258 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.424363 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.429064 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.429075 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.429089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.429122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.432000 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.432662 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.432670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.432672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.432798 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.433175 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.433179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.433180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.433541 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.433784 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.434082 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.434289 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":424,"threads_started":1}
W20260808 15:41:50.436058 24247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59170
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:41:50.490659 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.475758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.23.130.1:38305)
0808 15:41:50.476052 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.476059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.476347 (+ 288us) server_negotiation.cc:407] Connection header received
0808 15:41:50.476624 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.476632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.476729 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.476820 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.478246 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.479039 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.480241 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.480645 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.480735 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.485071 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.485076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.485079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.485114 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.488039 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.488687 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.488693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.488697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.488839 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.489222 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.489228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.489231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.489510 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.489757 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.490068 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.490287 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 15:41:50.492304 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24200
I20260808 15:41:50.505051 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.493263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.23.130.1:38305)
0808 15:41:50.493473 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.493477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.493851 (+ 374us) server_negotiation.cc:407] Connection header received
0808 15:41:50.493964 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.493967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.494023 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.494099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.495527 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.496233 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.497497 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.497923 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.498014 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.500367 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.500371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.500374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.500402 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.502941 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.503551 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.503554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.503556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.503679 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.503985 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.503988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.503990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.504219 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.504401 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.504656 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.504800 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:41:50.505748 24339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:50.590775 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24200
I20260808 15:41:50.609403 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24350
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:41:50.625339 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.610363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.23.130.2:40347)
0808 15:41:50.611029 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.611036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.611062 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:41:50.611123 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.611129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.611333 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.611469 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.612668 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.613691 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.615222 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.615794 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.615916 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.620007 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.620018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.620031 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.620070 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.622890 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.623507 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.623515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.623518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.623642 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.624034 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.624037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.624039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.624355 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.624579 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.624837 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.625039 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":322,"threads_started":1}
I20260808 15:41:50.626626 24490 generic_service.cc:196] Checking for leaks (request via RPC)
W20260808 15:41:50.633364 24167 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.23.130.62:37797
I20260808 15:41:50.711442 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24350
I20260808 15:41:50.729153 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24501
Aug 08 15:41:50 dist-test-slave-gp6n krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:41:50.745976 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:50.730444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.23.130.3:37797)
0808 15:41:50.731116 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:50.731123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:50.731154 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:41:50.731213 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:50.731219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:50.731473 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:50.731650 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:50.732945 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.733986 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.735344 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:50.735916 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:50.736058 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:50.740232 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:50.740243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:50.740257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:50.740294 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:50.743468 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.744086 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.744093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.744095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.744221 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:50.744550 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:50.744553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:50.744555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:50.744928 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:50.745174 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:50.745464 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 15:41:50.745676 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
I20260808 15:41:50.747521 24640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:50.831817 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24501
I20260808 15:41:50.850590 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24097
I20260808 15:41:50.851809 24166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:50.954921 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24097
2026-08-08T15:41:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4151 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:41:51 dist-test-slave-gp6n krb5kdc[24670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:41:51 dist-test-slave-gp6n krb5kdc[24670](info): set up 2 sockets
Aug 08 15:41:51 dist-test-slave-gp6n krb5kdc[24670](info): commencing operation
krb5kdc: starting...
W20260808 15:41:52.171185 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:41:52 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:41:52.379655 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37435
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:52.704075 24685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:52.704442 24685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:52.704605 24685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:52.711967 24685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:41:52.712126 24685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:52.712218 24685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:52.712308 24685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:41:52.712405 24685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:41:52.722143 24685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:52.723508 24685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:52.724856 24685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:52.736835 24692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:52.736958 24691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:52.736883 24694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:52.737650 24685 server_base.cc:1034] running on GCE node
I20260808 15:41:52.738493 24685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:52.739845 24685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:52.741096 24685 hybrid_clock.cc:648] HybridClock initialized: now 1786203712741061 us; error 55 us; skew 500 ppm
I20260808 15:41:52.741454 24685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:52 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203712, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:52.749150 24685 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:41:52.750689 24685 webserver.cc:460] Webserver started at http://127.23.130.62:44605/ using document root <none> and password file <none>
I20260808 15:41:52.751346 24685 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:52.751449 24685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:52.751711 24685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:52.754694 24685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "a53ccaf6de4a4c47b918a322b644871f"
format_stamp: "Formatted at 2026-08-08 15:41:52 on dist-test-slave-gp6n"
server_key: "0f705df7f90e18016b6a99ca2f722c4b"
server_key_iv: "7582ff18eee88304fcd843a3deb45c2a"
server_key_version: "encryptionkey@0"
I20260808 15:41:52.755434 24685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "a53ccaf6de4a4c47b918a322b644871f"
format_stamp: "Formatted at 2026-08-08 15:41:52 on dist-test-slave-gp6n"
server_key: "0f705df7f90e18016b6a99ca2f722c4b"
server_key_iv: "7582ff18eee88304fcd843a3deb45c2a"
server_key_version: "encryptionkey@0"
I20260808 15:41:52.760526 24685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:41:52.763911 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:52.765223 24685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:41:52.765401 24685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "a53ccaf6de4a4c47b918a322b644871f"
format_stamp: "Formatted at 2026-08-08 15:41:52 on dist-test-slave-gp6n"
server_key: "0f705df7f90e18016b6a99ca2f722c4b"
server_key_iv: "7582ff18eee88304fcd843a3deb45c2a"
server_key_version: "encryptionkey@0"
I20260808 15:41:52.765609 24685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:52.781229 24685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:52.785178 24685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:52.785477 24685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:52.804174 24685 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37435
I20260808 15:41:52.804198 24760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37435 every 8 connection(s)
I20260808 15:41:52.805605 24685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:41:52.808468 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24685
I20260808 15:41:52.808735 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:41:52.809434 24072 external_mini_cluster.cc:1428] Setting key 255a77ddd324322b4140b3e005580661
I20260808 15:41:52.809517 24761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:52.824404 24761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f: Bootstrap starting.
Aug 08 15:41:52 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:52.828377 24761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:52.829815 24761 log.cc:824] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:52.833408 24761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f: No bootstrap required, opened a new log
I20260808 15:41:52.835189 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:52.811473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.23.130.62:37435)
0808 15:41:52.812344 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:52.812360 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:52.812418 (+ 58us) server_negotiation.cc:407] Connection header received
0808 15:41:52.814667 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:52.814697 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:52.815222 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:52.815687 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:52.817053 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:52.819069 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:52.820352 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:52.821497 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:52.821690 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:52.826244 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:52.826257 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:52.826279 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:52.826323 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:52.830384 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:52.831355 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:52.831366 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:52.831375 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:52.831586 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:52.831991 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:52.831996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:52.831999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:52.832673 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:52.832958 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:52.833384 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 15:41:52.833843 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":377,"threads_started":1}
I20260808 15:41:52.841903 24761 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } }
I20260808 15:41:52.842259 24761 raft_consensus.cc:374] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:52.842324 24761 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53ccaf6de4a4c47b918a322b644871f, State: Initialized, Role: FOLLOWER
I20260808 15:41:52.842941 24761 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } }
I20260808 15:41:52.843187 24761 raft_consensus.cc:388] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:41:52.843325 24761 raft_consensus.cc:482] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:41:52.843504 24761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:52.845993 24761 raft_consensus.cc:504] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } }
I20260808 15:41:52.846565 24761 leader_election.cc:302] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53ccaf6de4a4c47b918a322b644871f; no voters:
I20260808 15:41:52.847164 24761 leader_election.cc:288] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:41:52.847268 24767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:52.848024 24767 raft_consensus.cc:686] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [term 1 LEADER]: Becoming Leader. State: Replica: a53ccaf6de4a4c47b918a322b644871f, State: Running, Role: LEADER
I20260808 15:41:52.848686 24767 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } }
I20260808 15:41:52.849578 24761 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:41:52.851847 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53ccaf6de4a4c47b918a322b644871f. Latest consensus state: current_term: 1 leader_uuid: "a53ccaf6de4a4c47b918a322b644871f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } } }
I20260808 15:41:52.851892 24768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a53ccaf6de4a4c47b918a322b644871f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53ccaf6de4a4c47b918a322b644871f" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37435 } } }
I20260808 15:41:52.852288 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:52.852366 24768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:52.856612 24774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:41:52.860950 24774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:41:52.869098 24774 catalog_manager.cc:1347] Generated new cluster ID: 638d7ec530f245d78aabb700e35aba93
I20260808 15:41:52.869217 24774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:41:52.881866 24774 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:41:52.883695 24774 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:41:52.897725 24774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a53ccaf6de4a4c47b918a322b644871f: Generated new TSK 0
I20260808 15:41:52.898658 24774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:41:52.951654 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:53.289680 24788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:53.289963 24788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:53.290063 24788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:53.297569 24788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:53.297701 24788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:53.297835 24788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:41:53.308423 24788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:53.309691 24788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:53.311117 24788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:53.324616 24795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:53.324522 24798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:53.324554 24796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:53.325811 24788 server_base.cc:1034] running on GCE node
I20260808 15:41:53.326632 24788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:53.327623 24788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:53.328897 24788 hybrid_clock.cc:648] HybridClock initialized: now 1786203713328828 us; error 51 us; skew 500 ppm
I20260808 15:41:53.329263 24788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:53 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203713, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:53.332933 24788 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:41:53.334388 24788 webserver.cc:460] Webserver started at http://127.23.130.1:44271/ using document root <none> and password file <none>
I20260808 15:41:53.335142 24788 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:53.335279 24788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:53.335608 24788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:53.338686 24788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "6a57288e2846433498e4daa972e5f56b"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "05e72730fb3b30107f456742e3f4e952"
server_key_iv: "cfd08745f62bc60544b09ab871e4f39c"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.339493 24788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "6a57288e2846433498e4daa972e5f56b"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "05e72730fb3b30107f456742e3f4e952"
server_key_iv: "cfd08745f62bc60544b09ab871e4f39c"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.344596 24788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:41:53.347999 24806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.349442 24788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:41:53.349644 24788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "6a57288e2846433498e4daa972e5f56b"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "05e72730fb3b30107f456742e3f4e952"
server_key_iv: "cfd08745f62bc60544b09ab871e4f39c"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.349892 24788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:53.367336 24788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:53.371765 24788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:53.372094 24788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:53.373128 24788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:53.374622 24788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:53.374737 24788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.374914 24788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:53.375010 24788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.412861 24788 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:46027
I20260808 15:41:53.412909 24936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:46027 every 8 connection(s)
I20260808 15:41:53.414242 24788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:41:53.422353 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24788
I20260808 15:41:53.422676 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:41:53.423542 24072 external_mini_cluster.cc:1428] Setting key 2fcd0d1ad1111a3a556f4d68c9dec378
Aug 08 15:41:53 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203713, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:53.437664 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:53.417195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:46125 (local address 127.23.130.62:37435)
0808 15:41:53.417735 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:53.417742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:53.418795 (+ 1053us) server_negotiation.cc:407] Connection header received
0808 15:41:53.420745 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:53.420750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:53.420835 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:53.420943 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:53.422620 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.423905 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.425965 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.426720 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.426893 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:53.430626 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:53.430636 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:53.430645 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:53.430693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:53.433886 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:53.434968 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:53.434972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:53.434975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:53.435105 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:53.435667 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:53.435671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:53.435673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:53.435960 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:53.436076 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:53.437113 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0808 15:41:53.437331 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":244,"threads_started":1}
I20260808 15:41:53.439492 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:53.417756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37435 (local address 127.23.130.1:46125)
0808 15:41:53.418622 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:53.418668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:53.420477 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:53.421182 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:53.421190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:53.421757 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:53.422367 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:53.422386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.424105 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.424110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:53.425789 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:53.425802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.426959 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.426964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:53.427173 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:53.427941 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:53.427964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:53.430336 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:53.434045 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:53.434058 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:53.434078 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:53.434810 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:53.435234 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:53.435243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:53.435250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:53.435527 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:53.436101 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:53.436113 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:53.436618 (+ 505us) client_negotiation.cc:769] Sending connection context
0808 15:41:53.436987 (+ 369us) client_negotiation.cc:240] Negotiation successful
0808 15:41:53.437659 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
I20260808 15:41:53.441457 24937 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37435
I20260808 15:41:53.441809 24937 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:53.442656 24937 heartbeater.cc:511] Master 127.23.130.62:37435 requested a full tablet report, sending...
I20260808 15:41:53.444854 24720 ts_manager.cc:194] Registered new tserver with Master: 6a57288e2846433498e4daa972e5f56b (127.23.130.1:46027)
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
I20260808 15:41:53.446506 24720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:46125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:41:53.473728 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:53.803155 24943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:53.803458 24943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:53.803689 24943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:53.811028 24943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:53.811137 24943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:53.811376 24943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:41:53.821238 24943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:53.822721 24943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:53.824096 24943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:53.837720 24948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:53.837805 24949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:53.838351 24953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:53.844120 24943 server_base.cc:1034] running on GCE node
I20260808 15:41:53.844823 24943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:53.845840 24943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:53.847096 24943 hybrid_clock.cc:648] HybridClock initialized: now 1786203713847050 us; error 40 us; skew 500 ppm
I20260808 15:41:53.847473 24943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:53 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203713, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:53.850895 24943 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:41:53.852550 24943 webserver.cc:460] Webserver started at http://127.23.130.2:35067/ using document root <none> and password file <none>
I20260808 15:41:53.853363 24943 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:53.853546 24943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:53.853945 24943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:53.857247 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "65be63cb9d2c40ab8b7328162901d6c4"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "dd8d6aff8cc99f371daf0f1385011a97"
server_key_iv: "1079b5aea22e82ab2b3abb100b958776"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.858076 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "65be63cb9d2c40ab8b7328162901d6c4"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "dd8d6aff8cc99f371daf0f1385011a97"
server_key_iv: "1079b5aea22e82ab2b3abb100b958776"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.863606 24943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:41:53.867285 24960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.868731 24943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:41:53.868919 24943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "65be63cb9d2c40ab8b7328162901d6c4"
format_stamp: "Formatted at 2026-08-08 15:41:53 on dist-test-slave-gp6n"
server_key: "dd8d6aff8cc99f371daf0f1385011a97"
server_key_iv: "1079b5aea22e82ab2b3abb100b958776"
server_key_version: "encryptionkey@0"
I20260808 15:41:53.869133 24943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:53.884397 24943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:53.888732 24943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:53.889062 24943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:53.890131 24943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:53.891575 24943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:53.891690 24943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.891851 24943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:53.891947 24943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:53.926707 24943 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:34009
I20260808 15:41:53.926750 25083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:34009 every 8 connection(s)
I20260808 15:41:53.928071 24943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:41:53.934329 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 24943
I20260808 15:41:53.934599 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:41:53.935108 24072 external_mini_cluster.cc:1428] Setting key f7a740d5a6e3b51d37852539af2b30bd
Aug 08 15:41:53 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203713, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:53.951145 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:53.930826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:45171 (local address 127.23.130.62:37435)
0808 15:41:53.931003 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:53.931008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:53.932269 (+ 1261us) server_negotiation.cc:407] Connection header received
0808 15:41:53.934123 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:53.934129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:53.934228 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:53.934347 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:53.936231 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.937023 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.939795 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.940252 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.940353 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:53.944667 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:53.944672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:53.944676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:53.944709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:53.947413 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:53.948449 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:53.948453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:53.948455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:53.948591 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:53.949113 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:53.949116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:53.949118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:53.949409 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:53.949595 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:53.950669 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0808 15:41:53.950886 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:41:53.952742 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:53.931293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37435 (local address 127.23.130.2:45171)
0808 15:41:53.932059 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:53.932099 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:53.933819 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:53.934649 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:53.934658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:53.935279 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:53.936049 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:53.936071 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.937201 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.937206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:53.939631 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:53.939652 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:53.940505 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:53.940516 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:53.940867 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:53.941637 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:53.941672 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:53.944410 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:53.947579 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:53.947591 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:53.947611 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:53.948306 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:53.948710 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:53.948718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:53.948724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:53.948992 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:53.949547 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:53.949559 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:53.950141 (+ 582us) client_negotiation.cc:769] Sending connection context
0808 15:41:53.950481 (+ 340us) client_negotiation.cc:240] Negotiation successful
0808 15:41:53.951114 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":294,"threads_started":1}
I20260808 15:41:53.954535 25085 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37435
I20260808 15:41:53.954918 25085 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:53.955747 25085 heartbeater.cc:511] Master 127.23.130.62:37435 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:41:53.957296 24720 ts_manager.cc:194] Registered new tserver with Master: 65be63cb9d2c40ab8b7328162901d6c4 (127.23.130.2:34009)
I20260808 15:41:53.958205 24720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:45171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:41:53.987031 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:54.335106 25091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:54.335390 25091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:54.335554 25091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:54.342998 25091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:54.343111 25091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:54.343289 25091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:41:54.353550 25091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:54.355127 25091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:54.356575 25091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:54.370645 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:54.370599 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:54.370893 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:54.371210 25091 server_base.cc:1034] running on GCE node
I20260808 15:41:54.371914 25091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:54.373095 25091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:54.374336 25091 hybrid_clock.cc:648] HybridClock initialized: now 1786203714374299 us; error 61 us; skew 500 ppm
I20260808 15:41:54.374707 25091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:54.378008 25091 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:41:54.379621 25091 webserver.cc:460] Webserver started at http://127.23.130.3:44759/ using document root <none> and password file <none>
I20260808 15:41:54.380347 25091 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:54.380512 25091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:54.380843 25091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:54.383984 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "f01e2f14936e45c594b9e770db4735ff"
format_stamp: "Formatted at 2026-08-08 15:41:54 on dist-test-slave-gp6n"
server_key: "0436990b5ad3de89736cd1482a3efd58"
server_key_iv: "ac4df867cb206aeff62ae494f09239c2"
server_key_version: "encryptionkey@0"
I20260808 15:41:54.384953 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "f01e2f14936e45c594b9e770db4735ff"
format_stamp: "Formatted at 2026-08-08 15:41:54 on dist-test-slave-gp6n"
server_key: "0436990b5ad3de89736cd1482a3efd58"
server_key_iv: "ac4df867cb206aeff62ae494f09239c2"
server_key_version: "encryptionkey@0"
I20260808 15:41:54.390524 25091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:41:54.394126 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:54.395605 25091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:41:54.395792 25091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "f01e2f14936e45c594b9e770db4735ff"
format_stamp: "Formatted at 2026-08-08 15:41:54 on dist-test-slave-gp6n"
server_key: "0436990b5ad3de89736cd1482a3efd58"
server_key_iv: "ac4df867cb206aeff62ae494f09239c2"
server_key_version: "encryptionkey@0"
I20260808 15:41:54.396025 25091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:54.410466 25091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:54.415684 25091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:54.416011 25091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:54.417191 25091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:41:54.418905 25091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:54.419003 25091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:54.419111 25091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:54.419178 25091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:54.449720 24937 heartbeater.cc:503] Master 127.23.130.62:37435 was elected leader, sending a full tablet report...
I20260808 15:41:54.456070 25091 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:43169
I20260808 15:41:54.456130 25237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:43169 every 8 connection(s)
I20260808 15:41:54.457474 25091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:41:54.458220 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 25091
I20260808 15:41:54.458472 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:41:54.459043 24072 external_mini_cluster.cc:1428] Setting key 2e1cb32170f9f4a35946fb620014d772
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:54.479352 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.460554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:34341 (local address 127.23.130.62:37435)
0808 15:41:54.461247 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.461254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.462293 (+ 1039us) server_negotiation.cc:407] Connection header received
0808 15:41:54.464224 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.464228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.464293 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.464385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.465932 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.466823 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.468368 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.468758 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.468859 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.472586 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.472593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.472598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.472658 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.476344 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.477171 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.477175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.477177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.477313 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.477747 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.477752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.477755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.478038 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.478152 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.478912 (+ 760us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.479078 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":262,"threads_started":1}
I20260808 15:41:54.480352 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.461292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37435 (local address 127.23.130.3:34341)
0808 15:41:54.462125 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.462173 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.463967 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.464621 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.464629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.465162 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:54.465774 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.465793 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.466950 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.466954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.468227 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.468238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.468955 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.468961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.469204 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.469959 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:54.469982 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:54.472339 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:54.476542 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:54.476553 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:54.476573 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:54.477049 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:54.477423 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:54.477427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:54.477429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:54.477631 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:54.478152 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:54.478159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:54.478561 (+ 402us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.478800 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.479212 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260808 15:41:54.481752 25238 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37435
I20260808 15:41:54.482105 25238 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:54.482949 25238 heartbeater.cc:511] Master 127.23.130.62:37435 requested a full tablet report, sending...
I20260808 15:41:54.484486 24720 ts_manager.cc:194] Registered new tserver with Master: f01e2f14936e45c594b9e770db4735ff (127.23.130.3:43169)
I20260808 15:41:54.485553 24720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:34341
I20260808 15:41:54.487735 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:41:54.506995 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.493952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.23.130.62:37435)
0808 15:41:54.494127 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.494132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.494783 (+ 651us) server_negotiation.cc:407] Connection header received
0808 15:41:54.494929 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.494932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.494995 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.495078 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.496563 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.497280 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.498851 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.499279 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.499446 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.501885 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.501891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.501901 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.501938 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.504300 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.504949 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.504955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.504959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.505084 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.505472 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.505478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.505482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.505813 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.506089 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.506431 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.506667 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:41:54.513262 24720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:41:54.520184 24720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:41:54.559537 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.533891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.23.130.1:46027)
0808 15:41:54.548801 (+ 14910us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.548815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.550493 (+ 1678us) server_negotiation.cc:407] Connection header received
0808 15:41:54.550637 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.550645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.550890 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.551112 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:54.552382 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.554106 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.556847 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.558270 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.558472 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.558734 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.559016 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.559184 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14671,"thread_start_us":14478,"threads_started":1}
I20260808 15:41:54.559659 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.533348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46027 (local address 127.0.0.1:59052)
0808 15:41:54.550301 (+ 16953us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.550323 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.550495 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.551391 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.551396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.551437 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:54.552136 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.552154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.554338 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.554343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.556636 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.556648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.558528 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.558535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.558729 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.558787 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.558969 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.559178 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16801,"mutex_wait_us":123,"thread_start_us":16619,"threads_started":1}
I20260808 15:41:54.560177 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.532727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34009 (local address 127.0.0.1:47850)
0808 15:41:54.547615 (+ 14888us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.547641 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.547806 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.551129 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.551138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.551178 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:54.551768 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.551789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.553839 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.553844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.556590 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.556624 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.559030 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.559038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.559306 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.559454 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.559548 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.559700 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14712,"mutex_wait_us":143,"thread_start_us":10661,"threads_started":1}
I20260808 15:41:54.560606 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.532526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:43169 (local address 127.0.0.1:52614)
0808 15:41:54.543793 (+ 11267us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.543841 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.544035 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.551453 (+ 7418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.551459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.551486 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:54.552196 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.552209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.554212 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.554217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.556995 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.557007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.559045 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.559050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.559305 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.559451 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.559548 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.560191 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11018,"mutex_wait_us":709,"spinlock_wait_cycles":1408,"thread_start_us":15390,"threads_started":1}
I20260808 15:41:54.561110 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.533246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.23.130.2:34009)
0808 15:41:54.549453 (+ 16207us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.549462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.549504 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:41:54.549635 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.549648 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.550823 (+ 1175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.551122 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:54.551993 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.553646 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.556995 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.558518 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.558652 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.558836 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.560598 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.560784 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15979,"thread_start_us":15829,"threads_started":1}
I20260808 15:41:54.561282 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.532806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.23.130.3:43169)
0808 15:41:54.550785 (+ 17979us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.550793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.550825 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:41:54.550911 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.550931 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.551194 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.551427 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:54.552424 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.554059 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.557173 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.558533 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.558652 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.558872 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.559655 (+ 783us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.560967 (+ 1312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17813,"thread_start_us":17644,"threads_started":1}
I20260808 15:41:54.564112 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce05d7bb24561be406f7155b3674b (DEFAULT_TABLE table=test-table [id=4ebd69d7f10745d49384566b2c8dc9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:54.564738 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce05d7bb24561be406f7155b3674b (DEFAULT_TABLE table=test-table [id=4ebd69d7f10745d49384566b2c8dc9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:54.566367 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce05d7bb24561be406f7155b3674b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:54.569160 25166 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce05d7bb24561be406f7155b3674b (DEFAULT_TABLE table=test-table [id=4ebd69d7f10745d49384566b2c8dc9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:54.570736 25166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce05d7bb24561be406f7155b3674b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:54.578944 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce05d7bb24561be406f7155b3674b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:54.581830 25261 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Bootstrap starting.
I20260808 15:41:54.585964 25261 tablet_bootstrap.cc:654] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:54.587286 25261 log.cc:824] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:54.590518 25262 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Bootstrap starting.
I20260808 15:41:54.590835 25261 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: No bootstrap required, opened a new log
I20260808 15:41:54.591113 25261 ts_tablet_manager.cc:1397] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 15:41:54.593395 25262 tablet_bootstrap.cc:654] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:54.594007 25264 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Bootstrap starting.
I20260808 15:41:54.594879 25262 log.cc:824] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:54.597851 25264 tablet_bootstrap.cc:654] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:54.599151 25264 log.cc:824] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:54.599234 25262 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: No bootstrap required, opened a new log
I20260808 15:41:54.599494 25262 ts_tablet_manager.cc:1397] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260808 15:41:54.601049 25261 raft_consensus.cc:348] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.601500 25261 raft_consensus.cc:374] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:54.601615 25261 raft_consensus.cc:729] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65be63cb9d2c40ab8b7328162901d6c4, State: Initialized, Role: FOLLOWER
I20260808 15:41:54.602224 25261 consensus_queue.cc:260] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.602440 25264 tablet_bootstrap.cc:492] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: No bootstrap required, opened a new log
I20260808 15:41:54.602730 25264 ts_tablet_manager.cc:1397] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260808 15:41:54.603403 25085 heartbeater.cc:503] Master 127.23.130.62:37435 was elected leader, sending a full tablet report...
I20260808 15:41:54.603928 25261 ts_tablet_manager.cc:1428] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:41:54.608778 25262 raft_consensus.cc:348] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.609404 25262 raft_consensus.cc:374] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:54.609517 25262 raft_consensus.cc:729] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01e2f14936e45c594b9e770db4735ff, State: Initialized, Role: FOLLOWER
I20260808 15:41:54.610296 25262 consensus_queue.cc:260] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.611485 25238 heartbeater.cc:503] Master 127.23.130.62:37435 was elected leader, sending a full tablet report...
I20260808 15:41:54.612058 25262 ts_tablet_manager.cc:1428] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260808 15:41:54.613734 25264 raft_consensus.cc:348] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.614209 25264 raft_consensus.cc:374] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:54.614327 25264 raft_consensus.cc:729] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a57288e2846433498e4daa972e5f56b, State: Initialized, Role: FOLLOWER
I20260808 15:41:54.614982 25264 consensus_queue.cc:260] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.616585 25264 ts_tablet_manager.cc:1428] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
W20260808 15:41:54.667128 24938 tablet.cc:2369] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:41:54.680281 25086 tablet.cc:2369] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:41:54.702929 25267 raft_consensus.cc:482] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:41:54.703306 25267 raft_consensus.cc:504] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.705040 25267 leader_election.cc:288] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f01e2f14936e45c594b9e770db4735ff (127.23.130.3:43169), 6a57288e2846433498e4daa972e5f56b (127.23.130.1:46027)
W20260808 15:41:54.709411 25239 tablet.cc:2369] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:41:54.713573 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.705588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:54209 (local address 127.23.130.1:46027)
0808 15:41:54.705833 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.705841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.706773 (+ 932us) server_negotiation.cc:407] Connection header received
0808 15:41:54.706989 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.706995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.707082 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.707194 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:54.708102 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.709408 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.711700 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.712507 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.712844 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.712925 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.713176 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.713309 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:41:54.713596 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.705501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:47101 (local address 127.23.130.3:43169)
0808 15:41:54.705742 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.705749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.706832 (+ 1083us) server_negotiation.cc:407] Connection header received
0808 15:41:54.706989 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.706995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.707082 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.707194 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:54.708102 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.709472 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.711721 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.712544 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.712745 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.712833 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.713188 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.713326 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:41:54.713825 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.705457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:43169 (local address 127.23.130.2:47101)
0808 15:41:54.706716 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.706732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.706838 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.707234 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.707240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.707289 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:54.707916 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.707925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.709652 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.709659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.711541 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.711555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.712810 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.712816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.713016 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.713042 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.713159 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.713300 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":243,"spinlock_wait_cycles":12416,"thread_start_us":610,"threads_started":1}
I20260808 15:41:54.713867 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.705451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46027 (local address 127.23.130.2:54209)
0808 15:41:54.706638 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:54.706660 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:54.706833 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:54.707234 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:54.707240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:54.707287 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:54.707887 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.707907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.709580 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.709586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.711521 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:54.711535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.712790 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.712799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:54.713000 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.713027 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:41:54.713137 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:41:54.713278 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":34,"spinlock_wait_cycles":15104,"thread_start_us":455,"threads_started":1}
I20260808 15:41:54.714856 25188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce05d7bb24561be406f7155b3674b" candidate_uuid: "65be63cb9d2c40ab8b7328162901d6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01e2f14936e45c594b9e770db4735ff" is_pre_election: true
I20260808 15:41:54.714941 24887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce05d7bb24561be406f7155b3674b" candidate_uuid: "65be63cb9d2c40ab8b7328162901d6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a57288e2846433498e4daa972e5f56b" is_pre_election: true
I20260808 15:41:54.715415 25188 raft_consensus.cc:2454] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65be63cb9d2c40ab8b7328162901d6c4 in term 0.
I20260808 15:41:54.715489 24887 raft_consensus.cc:2454] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65be63cb9d2c40ab8b7328162901d6c4 in term 0.
I20260808 15:41:54.716198 24961 leader_election.cc:302] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 65be63cb9d2c40ab8b7328162901d6c4, 6a57288e2846433498e4daa972e5f56b, f01e2f14936e45c594b9e770db4735ff; no voters:
I20260808 15:41:54.716614 25267 raft_consensus.cc:2781] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:41:54.716773 25267 raft_consensus.cc:482] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:41:54.716895 25267 raft_consensus.cc:3037] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:54.719331 25267 raft_consensus.cc:504] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.719998 25267 leader_election.cc:288] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [CANDIDATE]: Term 1 election: Requested vote from peers f01e2f14936e45c594b9e770db4735ff (127.23.130.3:43169), 6a57288e2846433498e4daa972e5f56b (127.23.130.1:46027)
I20260808 15:41:54.720480 25188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce05d7bb24561be406f7155b3674b" candidate_uuid: "65be63cb9d2c40ab8b7328162901d6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01e2f14936e45c594b9e770db4735ff"
I20260808 15:41:54.720513 24887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce05d7bb24561be406f7155b3674b" candidate_uuid: "65be63cb9d2c40ab8b7328162901d6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a57288e2846433498e4daa972e5f56b"
I20260808 15:41:54.720692 25188 raft_consensus.cc:3037] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:54.720723 24887 raft_consensus.cc:3037] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:54.723105 25188 raft_consensus.cc:2454] T 3acce05d7bb24561be406f7155b3674b P f01e2f14936e45c594b9e770db4735ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65be63cb9d2c40ab8b7328162901d6c4 in term 1.
I20260808 15:41:54.723359 24887 raft_consensus.cc:2454] T 3acce05d7bb24561be406f7155b3674b P 6a57288e2846433498e4daa972e5f56b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65be63cb9d2c40ab8b7328162901d6c4 in term 1.
I20260808 15:41:54.723634 24961 leader_election.cc:302] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65be63cb9d2c40ab8b7328162901d6c4, f01e2f14936e45c594b9e770db4735ff; no voters:
I20260808 15:41:54.723950 25267 raft_consensus.cc:2781] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:54.724406 25267 raft_consensus.cc:686] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [term 1 LEADER]: Becoming Leader. State: Replica: 65be63cb9d2c40ab8b7328162901d6c4, State: Running, Role: LEADER
I20260808 15:41:54.724982 25267 consensus_queue.cc:237] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } }
I20260808 15:41:54.729745 24719 catalog_manager.cc:5495] T 3acce05d7bb24561be406f7155b3674b P 65be63cb9d2c40ab8b7328162901d6c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65be63cb9d2c40ab8b7328162901d6c4 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "65be63cb9d2c40ab8b7328162901d6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01e2f14936e45c594b9e770db4735ff" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65be63cb9d2c40ab8b7328162901d6c4" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a57288e2846433498e4daa972e5f56b" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46027 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:41:54.755827 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.738739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.23.130.1:46027)
0808 15:41:54.739061 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.739071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.739434 (+ 363us) server_negotiation.cc:407] Connection header received
0808 15:41:54.739686 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.739692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.739762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.739908 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.741670 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.742633 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.744052 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.744543 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.744698 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.749354 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.749373 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.749390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.749431 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.752693 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.753507 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.753515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.753517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.753657 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.754136 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.754142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.754145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.754589 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.754939 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.755205 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.755428 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:41:54.822832 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.804503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.23.130.1:46027)
0808 15:41:54.804739 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.804747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.805547 (+ 800us) server_negotiation.cc:407] Connection header received
0808 15:41:54.805710 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.805715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.805797 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.805901 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.807794 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.809127 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.811029 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.811845 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.812013 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.817168 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.817175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.817180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.817224 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.820046 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.820704 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.820716 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.820719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.820860 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.821318 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.821327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.821329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.821601 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.821879 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.822172 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.822454 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:41:54 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:41:54.874739 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.860500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.23.130.1:46027)
0808 15:41:54.860827 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.860835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.860991 (+ 156us) server_negotiation.cc:407] Connection header received
0808 15:41:54.861150 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.861154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.861218 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.861303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.862739 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.863462 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.864624 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.865050 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.865206 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.869247 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.869257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.869262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.869317 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.871891 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.872683 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.872688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.872691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.872836 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.873172 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.873175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.873178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.873592 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.873807 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.874046 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.874242 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260808 15:41:54.876569 24841 tablet_service.cc:2908] Rejecting scan request for tablet 3acce05d7bb24561be406f7155b3674b: Uninitialized: clean time has not yet been initialized
I20260808 15:41:54.879911 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24788
I20260808 15:41:54.893039 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:54.880914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.23.130.1:46027)
0808 15:41:54.881092 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:54.881100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:54.881438 (+ 338us) server_negotiation.cc:407] Connection header received
0808 15:41:54.881642 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:54.881648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:54.881732 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:54.881840 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:54.883267 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.883948 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.885166 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:54.885626 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:54.885782 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:54.888094 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:54.888098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:54.888102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:54.888130 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:54.890764 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.891374 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.891378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.891380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.891495 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:54.891821 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:54.891824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:54.891826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:54.892094 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:54.892297 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:54.892594 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 15:41:54.892748 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:41:54.893759 24930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:54.985939 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24788
I20260808 15:41:55.007725 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24943
Aug 08 15:41:55 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:41:55.022568 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:55.008586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.23.130.2:34009)
0808 15:41:55.008804 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:55.008809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:55.008827 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:41:55.008974 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:55.008978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:55.009046 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:55.009131 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:55.010299 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:55.011114 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:55.012341 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:55.012718 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:55.012806 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:55.016979 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:55.016994 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:55.017009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:55.017047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:55.019821 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:55.020437 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:55.020444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:55.020446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:55.020593 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:55.020929 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:55.020948 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:55.020950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:55.021275 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:55.021495 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:55.021829 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 15:41:55.022157 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:41:55.023327 25079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:55.123217 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24943
I20260808 15:41:55.143994 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25091
Aug 08 15:41:55 dist-test-slave-gp6n krb5kdc[24670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:41:55.160585 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:55.144882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.23.130.3:43169)
0808 15:41:55.145060 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:55.145065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:55.145084 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:41:55.145162 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:55.145166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:55.145229 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:55.145322 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:55.146850 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:55.147612 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:55.149122 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:55.149591 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:55.149722 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:55.154577 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:55.154594 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:55.154610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:55.154655 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:55.157795 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:55.158529 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:55.158537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:55.158540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:55.158687 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:55.159058 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:55.159062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:55.159064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:55.159456 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:55.159688 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:55.159951 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 15:41:55.160240 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:41:55.161288 25233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:55.250461 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 25091
I20260808 15:41:55.270625 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 24685
I20260808 15:41:55.271938 24756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:41:55.381788 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 24685
2026-08-08T15:41:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4427 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:41:55 dist-test-slave-gp6n krb5kdc[25297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:41:55 dist-test-slave-gp6n krb5kdc[25297](info): set up 2 sockets
Aug 08 15:41:55 dist-test-slave-gp6n krb5kdc[25297](info): commencing operation
krb5kdc: starting...
W20260808 15:41:56.553237 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:41:56 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:41:56.759497 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33007
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:33007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:57.082103 25312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:57.082479 25312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:57.082659 25312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:57.089849 25312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:41:57.090020 25312 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:41:57.090332 25312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:57.090502 25312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:57.090613 25312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:41:57.090718 25312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:41:57.101516 25312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:33007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:57.102875 25312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:57.104261 25312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:57.116361 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:57.116451 25318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:57.116428 25317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:57.117255 25312 server_base.cc:1034] running on GCE node
I20260808 15:41:57.118165 25312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:57.119679 25312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:57.120949 25312 hybrid_clock.cc:648] HybridClock initialized: now 1786203717120890 us; error 71 us; skew 500 ppm
I20260808 15:41:57.121304 25312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:57 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203717, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:57.124673 25312 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:41:57.126129 25312 webserver.cc:460] Webserver started at http://127.23.130.62:43089/ using document root <none> and password file <none>
I20260808 15:41:57.126883 25312 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:57.126976 25312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:57.127240 25312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:57.130225 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "4329ae733cd6498fae3e2aa214d730bc"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "01d6daad64bd1638de7da0f61c19ce1a"
server_key_iv: "04d5f7825d355019fc714375584fb441"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.131028 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "4329ae733cd6498fae3e2aa214d730bc"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "01d6daad64bd1638de7da0f61c19ce1a"
server_key_iv: "04d5f7825d355019fc714375584fb441"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.136099 25312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:41:57.139541 25329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:57.140985 25312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:41:57.141165 25312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "4329ae733cd6498fae3e2aa214d730bc"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "01d6daad64bd1638de7da0f61c19ce1a"
server_key_iv: "04d5f7825d355019fc714375584fb441"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.141402 25312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:57.157752 25312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:57.162038 25312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:57.162317 25312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:57.185199 25312 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:33007
I20260808 15:41:57.185218 25399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:33007 every 8 connection(s)
I20260808 15:41:57.186720 25312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:41:57.189474 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 25312
I20260808 15:41:57.189756 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:41:57.190503 24072 external_mini_cluster.cc:1428] Setting key 2bfcf0874e973c12f4578adc3633e430
I20260808 15:41:57.191298 25400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:57.205766 25400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Bootstrap starting.
Aug 08 15:41:57 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:57.209956 25400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:57.211387 25400 log.cc:824] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:57.215176 25400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: No bootstrap required, opened a new log
I20260808 15:41:57.216503 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:57.192611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.23.130.62:33007)
0808 15:41:57.193429 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:57.193441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:57.193482 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:41:57.195649 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:57.195677 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:57.196215 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:57.196676 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:57.198061 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.200206 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.201519 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.202573 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.202765 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:57.207318 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:57.207335 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:57.207358 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:57.207417 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:57.211565 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.212532 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.212538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.212544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.212718 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.213188 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.213193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.213196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.213908 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:57.214198 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:57.214708 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 15:41:57.215154 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":396,"threads_started":1}
I20260808 15:41:57.223907 25400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } }
I20260808 15:41:57.224231 25400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:57.224303 25400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4329ae733cd6498fae3e2aa214d730bc, State: Initialized, Role: FOLLOWER
I20260808 15:41:57.224787 25400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } }
I20260808 15:41:57.224980 25400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:41:57.225111 25400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:41:57.225286 25400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:57.227770 25400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } }
I20260808 15:41:57.228258 25400 leader_election.cc:302] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4329ae733cd6498fae3e2aa214d730bc; no voters:
I20260808 15:41:57.228884 25400 leader_election.cc:288] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:41:57.229038 25405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:57.229902 25405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [term 1 LEADER]: Becoming Leader. State: Replica: 4329ae733cd6498fae3e2aa214d730bc, State: Running, Role: LEADER
I20260808 15:41:57.230556 25405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } }
I20260808 15:41:57.231153 25400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:41:57.233765 25407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4329ae733cd6498fae3e2aa214d730bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } } }
I20260808 15:41:57.233681 25408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4329ae733cd6498fae3e2aa214d730bc. Latest consensus state: current_term: 1 leader_uuid: "4329ae733cd6498fae3e2aa214d730bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4329ae733cd6498fae3e2aa214d730bc" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33007 } } }
I20260808 15:41:57.234154 25407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:57.234233 25408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc [sys.catalog]: This master's current role is: LEADER
I20260808 15:41:57.236474 25413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:41:57.246280 25413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:41:57.255780 25413 catalog_manager.cc:1347] Generated new cluster ID: 963e2dfbfe5543fd900efe07921eb9bd
I20260808 15:41:57.255959 25413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:41:57.296595 25413 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:41:57.297971 25413 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:41:57.306718 25413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Generated new TSK 0
I20260808 15:41:57.307555 25413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:41:57.369740 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33007
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:57.703461 25430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:57.703794 25430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:57.703975 25430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:57.711252 25430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:57.711402 25430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:57.711553 25430 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:41:57.711687 25430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:41:57.721493 25430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:57.722872 25430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:57.724232 25430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:57.738229 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:57.738245 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:57.738940 25430 server_base.cc:1034] running on GCE node
W20260808 15:41:57.739020 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:57.739718 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:57.740644 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:57.741909 25430 hybrid_clock.cc:648] HybridClock initialized: now 1786203717741826 us; error 55 us; skew 500 ppm
I20260808 15:41:57.742223 25430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:57 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203717, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:57.745877 25430 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:41:57.747460 25430 webserver.cc:460] Webserver started at http://127.23.130.1:45405/ using document root <none> and password file <none>
I20260808 15:41:57.748163 25430 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:57.748291 25430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:57.748628 25430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:57.751689 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "a0cc5de5774c4b7fb05511c18638af3e"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "18f689ac5ed1c04aa3c2c14ec5333fa3"
server_key_iv: "a10c9c350a9b0a315635ea944ceb4a14"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.752525 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "a0cc5de5774c4b7fb05511c18638af3e"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "18f689ac5ed1c04aa3c2c14ec5333fa3"
server_key_iv: "a10c9c350a9b0a315635ea944ceb4a14"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.757781 25430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:41:57.761263 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:57.762646 25430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:41:57.762818 25430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "a0cc5de5774c4b7fb05511c18638af3e"
format_stamp: "Formatted at 2026-08-08 15:41:57 on dist-test-slave-gp6n"
server_key: "18f689ac5ed1c04aa3c2c14ec5333fa3"
server_key_iv: "a10c9c350a9b0a315635ea944ceb4a14"
server_key_version: "encryptionkey@0"
I20260808 15:41:57.763047 25430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:57.787503 25430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:57.792673 25430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:57.792968 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:57.796038 25430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:57.796155 25430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:57.796290 25430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:57.796417 25430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 15:41:57 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203717, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:57.820220 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:57.796519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:57077 (local address 127.23.130.62:33007)
0808 15:41:57.797382 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:57.797387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:57.798166 (+ 779us) server_negotiation.cc:407] Connection header received
0808 15:41:57.800238 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:57.800246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:57.800345 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:57.800483 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:57.802725 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.803824 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.806460 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.807330 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.807481 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:57.813065 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:57.813072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:57.813079 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:57.813126 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:57.816448 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.817518 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.817525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.817529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.817703 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.818267 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.818273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.818276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.818708 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:57.818887 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:57.819617 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 15:41:57.819856 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":651,"threads_started":1}
I20260808 15:41:57.821336 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:57.797154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.1:57077)
0808 15:41:57.797989 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:57.798028 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:57.799942 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:57.800755 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:57.800769 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:57.801580 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:57.802540 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:57.802560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.803998 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.804010 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:57.806170 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:57.806214 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.807599 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.807605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:57.807824 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:57.808717 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:57.808754 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:57.812757 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:57.816616 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:57.816624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:57.816646 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:57.817363 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:57.817852 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:57.817859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:57.817865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:57.818116 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:57.818863 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:57.818871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:57.819236 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 15:41:57.819521 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 15:41:57.820052 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260808 15:41:57.844303 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:57.828275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.1:39043)
0808 15:41:57.829101 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:57.829126 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:57.829288 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:57.829731 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:57.829742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:57.830605 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:57.831158 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:57.831174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.832170 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.832178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:57.834389 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:57.834444 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.835356 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.835369 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:57.835628 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:57.836440 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:57.836469 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:57.837948 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:57.841189 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:57.841196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:57.841201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:57.841961 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:57.842459 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:57.842464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:57.842467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:57.842686 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:57.843452 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:57.843455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:57.843610 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 15:41:57.843728 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:41:57.843981 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":464,"threads_started":1}
I20260808 15:41:57.844738 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:57.827833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:39043 (local address 127.23.130.62:33007)
0808 15:41:57.828067 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:57.828075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:57.829228 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 15:41:57.829451 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:57.829457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:57.829550 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:57.829663 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:57.831307 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.832036 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.834596 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:57.835146 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:57.835297 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:57.838142 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:57.838147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:57.838150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:57.838181 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:57.841049 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.842115 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.842120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.842124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.842294 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:57.842852 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:57.842860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:57.842866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:57.843313 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:57.843505 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:57.843939 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 15:41:57.844251 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:41:57.855285 25430 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:41325
I20260808 15:41:57.855309 25578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:41325 every 8 connection(s)
I20260808 15:41:57.856524 25430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:41:57.858592 25579 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33007
I20260808 15:41:57.858853 25579 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:57.859510 25579 heartbeater.cc:511] Master 127.23.130.62:33007 requested a full tablet report, sending...
I20260808 15:41:57.861441 25349 ts_manager.cc:194] Registered new tserver with Master: a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325)
I20260808 15:41:57.862031 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 25430
I20260808 15:41:57.862262 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:41:57.862939 24072 external_mini_cluster.cc:1428] Setting key 32dca38674fbea6089e8eb64ef191589
I20260808 15:41:57.863152 25349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:57077
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:41:57.907797 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33007
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:58.265156 25583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:58.265432 25583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:58.265615 25583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:58.273353 25583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:58.273458 25583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:58.273628 25583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:41:58.273762 25583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:41:58.283304 25583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:58.284566 25583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:58.285913 25583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:58.299111 25592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:58.299268 25589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:58.299151 25590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:58.300369 25583 server_base.cc:1034] running on GCE node
I20260808 15:41:58.300979 25583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:58.301889 25583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:58.303153 25583 hybrid_clock.cc:648] HybridClock initialized: now 1786203718303072 us; error 51 us; skew 500 ppm
I20260808 15:41:58.303486 25583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:58.306856 25583 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:41:58.308269 25583 webserver.cc:460] Webserver started at http://127.23.130.2:33203/ using document root <none> and password file <none>
I20260808 15:41:58.308964 25583 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:58.309084 25583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:58.309458 25583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:58.312546 25583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "59545169461b491eb0c381a55eda42a8"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "18a1da99df2eaa69fe96bf882fb65620"
server_key_iv: "65282b93f1fe510e8855089cc0fa47f1"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.313391 25583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "59545169461b491eb0c381a55eda42a8"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "18a1da99df2eaa69fe96bf882fb65620"
server_key_iv: "65282b93f1fe510e8855089cc0fa47f1"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.318655 25583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:41:58.322082 25599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:58.323572 25583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:41:58.323752 25583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "59545169461b491eb0c381a55eda42a8"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "18a1da99df2eaa69fe96bf882fb65620"
server_key_iv: "65282b93f1fe510e8855089cc0fa47f1"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.324057 25583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:58.351253 25583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:58.355592 25583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:58.355839 25583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:58.358639 25583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:58.358750 25583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:58.358906 25583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:58.358989 25583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:58.383375 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.359328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:35287 (local address 127.23.130.62:33007)
0808 15:41:58.359952 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:58.359957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:58.361123 (+ 1166us) server_negotiation.cc:407] Connection header received
0808 15:41:58.363287 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:58.363293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:58.363385 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:58.363521 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:58.365914 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.367337 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.369989 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.370825 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.370980 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.375766 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:58.375775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:58.375781 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:58.375824 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:58.379236 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.380199 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.380205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.380208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.380330 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.380955 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.380962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.380967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.381308 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:58.381479 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:58.382614 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0808 15:41:58.382884 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":396,"threads_started":1}
I20260808 15:41:58.385249 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.360046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.2:35287)
0808 15:41:58.360946 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:58.360993 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:58.363006 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:58.363805 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:58.363824 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:58.364670 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:58.365653 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.365721 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.367546 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.367554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.369834 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.369846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.371079 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.371084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.371257 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.372033 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:58.372073 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:58.375451 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:58.379381 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.379391 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.379405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.380031 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.380517 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.380525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.380531 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.380814 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.381485 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:58.381496 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:58.382068 (+ 572us) client_negotiation.cc:769] Sending connection context
0808 15:41:58.382510 (+ 442us) client_negotiation.cc:240] Negotiation successful
0808 15:41:58.383254 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":418,"threads_started":1}
I20260808 15:41:58.408234 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.391412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:56179 (local address 127.23.130.62:33007)
0808 15:41:58.391601 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:58.391607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:58.392350 (+ 743us) server_negotiation.cc:407] Connection header received
0808 15:41:58.392528 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:58.392537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:58.392663 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:58.392801 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:58.393819 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.395339 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.396681 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.397568 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.397717 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.400423 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:58.400428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:58.400432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:58.400463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:58.404598 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.405721 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.405728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.405733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.405913 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.406454 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.406461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.406467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.406838 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:58.406959 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:58.407556 (+ 597us) server_negotiation.cc:299] Negotiation successful
0808 15:41:58.407843 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:41:58.408640 25583 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:44939
I20260808 15:41:58.408692 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.391530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.2:56179)
0808 15:41:58.392212 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:58.392231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:58.392356 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:58.392822 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:58.392825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:58.393244 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:58.393669 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.393680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.395495 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.395499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.396537 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.396548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.397798 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.397807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.398069 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.398822 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:58.398851 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:58.400182 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:58.404777 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.404784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.404790 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.405557 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.406058 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.406065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.406071 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.406280 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.406971 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:58.406976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:58.407253 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 15:41:58.407443 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 15:41:58.407932 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":386,"threads_started":1}
I20260808 15:41:58.408722 25736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:44939 every 8 connection(s)
I20260808 15:41:58.410425 25583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:41:58.413169 25737 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33007
I20260808 15:41:58.413432 25737 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:58.414000 25737 heartbeater.cc:511] Master 127.23.130.62:33007 requested a full tablet report, sending...
I20260808 15:41:58.415421 25349 ts_manager.cc:194] Registered new tserver with Master: 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939)
I20260808 15:41:58.416406 25349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:35287
I20260808 15:41:58.417812 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 25583
I20260808 15:41:58.418082 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:41:58.418715 24072 external_mini_cluster.cc:1428] Setting key 328bf0b3f5048043d4bc95a2059c7c0a
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:41:58.466208 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33007
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:41:58.795980 25741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:41:58.796320 25741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:41:58.796510 25741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:41:58.803645 25741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:41:58.803812 25741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:41:58.803982 25741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:41:58.804142 25741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:41:58.813964 25741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:41:58.815357 25741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:41:58.816772 25741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:41:58.830632 25746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:58.830627 25747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:41:58.830673 25749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:41:58.831431 25741 server_base.cc:1034] running on GCE node
I20260808 15:41:58.832124 25741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:41:58.833086 25741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:41:58.834306 25741 hybrid_clock.cc:648] HybridClock initialized: now 1786203718834251 us; error 25 us; skew 500 ppm
I20260808 15:41:58.834657 25741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:41:58.838032 25741 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:41:58.839443 25741 webserver.cc:460] Webserver started at http://127.23.130.3:46275/ using document root <none> and password file <none>
I20260808 15:41:58.840093 25741 fs_manager.cc:359] Metadata directory not provided
I20260808 15:41:58.840180 25741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:41:58.840447 25741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:41:58.843360 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "ddfbe2edf4b34ca985796186645243b2"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "8d5d627541fe9091a5d5722da5479443"
server_key_iv: "00480ba5e1ba5cc306064a449dfc39df"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.844123 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "ddfbe2edf4b34ca985796186645243b2"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "8d5d627541fe9091a5d5722da5479443"
server_key_iv: "00480ba5e1ba5cc306064a449dfc39df"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.849439 25741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:41:58.852902 25757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:58.854229 25741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:41:58.854414 25741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "ddfbe2edf4b34ca985796186645243b2"
format_stamp: "Formatted at 2026-08-08 15:41:58 on dist-test-slave-gp6n"
server_key: "8d5d627541fe9091a5d5722da5479443"
server_key_iv: "00480ba5e1ba5cc306064a449dfc39df"
server_key_version: "encryptionkey@0"
I20260808 15:41:58.854688 25741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:41:58.866199 25579 heartbeater.cc:503] Master 127.23.130.62:33007 was elected leader, sending a full tablet report...
I20260808 15:41:58.878926 25741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:41:58.883370 25741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:41:58.883664 25741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:41:58.886270 25741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:41:58.886394 25741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:41:58.886562 25741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:41:58.886653 25741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:58.910880 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.887236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:55431 (local address 127.23.130.62:33007)
0808 15:41:58.887445 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:58.887454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:58.889029 (+ 1575us) server_negotiation.cc:407] Connection header received
0808 15:41:58.891501 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:58.891508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:58.891586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:58.891647 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:58.894263 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.895635 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.897956 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.898387 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.898585 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.903382 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:58.903387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:58.903390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:58.903425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:58.907087 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.907867 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.907875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.907882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.908082 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.908532 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.908540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.908546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.909051 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:58.909247 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:58.910067 (+ 820us) server_negotiation.cc:299] Negotiation successful
0808 15:41:58.910386 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 15:41:58.912125 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.887835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.3:55431)
0808 15:41:58.888818 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:58.888876 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:58.891102 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:58.892223 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:58.892234 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:58.893052 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:58.894041 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.894074 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.895848 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.895855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.897787 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.897799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.898678 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.898685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.898894 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.899628 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:58.899675 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:58.903137 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:58.907250 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.907259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.907282 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.907735 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.908222 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.908226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.908228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.908415 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.909232 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:58.909240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:58.909667 (+ 427us) client_negotiation.cc:769] Sending connection context
0808 15:41:58.909903 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 15:41:58.910288 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":402,"threads_started":1}
I20260808 15:41:58.936584 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.919398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:49987 (local address 127.23.130.62:33007)
0808 15:41:58.919592 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:58.919598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:58.920440 (+ 842us) server_negotiation.cc:407] Connection header received
0808 15:41:58.920752 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:58.920760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:58.920835 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:58.920899 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:58.922556 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.923407 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.926100 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.926733 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.926884 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.929730 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:58.929737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:58.929747 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:58.929778 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:58.933326 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.934104 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.934107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.934109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.934235 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.934764 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.934771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.934775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:58.935158 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:58.935327 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:58.935931 (+ 604us) server_negotiation.cc:299] Negotiation successful
0808 15:41:58.936201 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:41:58.936981 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.919516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.23.130.3:49987)
0808 15:41:58.920301 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:58.920324 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:58.920583 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:58.921115 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:58.921127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:58.921843 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:58.922327 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.922338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.923565 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.923574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.925883 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:58.925905 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.927080 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.927093 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:58.927332 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.928206 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:58.928236 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:58.929519 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:58.933519 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.933528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.933533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.933969 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.934363 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:58.934368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:58.934372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:58.934598 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:58.935318 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:58.935329 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:58.935642 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 15:41:58.935845 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 15:41:58.936318 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":469,"threads_started":1}
I20260808 15:41:58.942235 25741 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:36691
I20260808 15:41:58.942277 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:36691 every 8 connection(s)
I20260808 15:41:58.943450 25741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:41:58.945521 25887 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33007
I20260808 15:41:58.945786 25887 heartbeater.cc:464] Registering TS with master...
I20260808 15:41:58.946529 25887 heartbeater.cc:511] Master 127.23.130.62:33007 requested a full tablet report, sending...
I20260808 15:41:58.946655 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 25741
I20260808 15:41:58.946869 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:41:58.947329 24072 external_mini_cluster.cc:1428] Setting key a777485f6bd4babb8fff58078f6dbe69
I20260808 15:41:58.948108 25349 ts_manager.cc:194] Registered new tserver with Master: ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691)
I20260808 15:41:58.949226 25349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:55431
I20260808 15:41:58.950640 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:41:58 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:59.001626 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:58.987632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.23.130.62:33007)
0808 15:41:58.987879 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:58.987886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:58.988086 (+ 200us) server_negotiation.cc:407] Connection header received
0808 15:41:58.988239 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:58.988243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:58.988312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:58.988400 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:58.989987 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.990789 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.992057 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:58.992438 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:58.992531 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:58.996466 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:58.996472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:58.996476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:58.996508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:58.999344 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:58.999920 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:58.999924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:58.999926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:59.000062 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:59.000408 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:59.000411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:59.000413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:59.000683 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:59.000900 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.001201 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.001372 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:41:59.006217 25347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:41:59.013352 25347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:41:59.040046 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.026904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.23.130.2:44939)
0808 15:41:59.028726 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.028739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.028778 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:41:59.028908 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.028920 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.029236 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.029488 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.030711 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.032718 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.036567 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.038473 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.038615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.038872 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.039550 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.039749 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":1551,"threads_started":1}
I20260808 15:41:59.040081 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.026479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44939 (local address 127.0.0.1:39882)
0808 15:41:59.028144 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.028161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.028259 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.029766 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.029773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.029821 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.030451 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.030470 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.032931 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.032940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.036313 (+ 3373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.036337 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.039018 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.039026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.039252 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.039308 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.039433 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.039624 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":593,"thread_start_us":773,"threads_started":1}
I20260808 15:41:59.043136 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.026707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.23.130.3:36691)
0808 15:41:59.029157 (+ 2450us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.029169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.029210 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:41:59.029353 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.029364 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.029677 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.032459 (+ 2782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.032889 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.036542 (+ 3653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.040910 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.042008 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.042179 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.042416 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.042654 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.042809 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2221,"thread_start_us":2014,"threads_started":1}
I20260808 15:41:59.043123 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.026473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36691 (local address 127.0.0.1:42752)
0808 15:41:59.027265 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.027315 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.027507 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.029976 (+ 2469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.029985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.030028 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.032138 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.032160 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.036734 (+ 4574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.036741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.040697 (+ 3956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.040710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.042226 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.042232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.042440 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.042476 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.042590 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.042727 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":323,"threads_started":1}
I20260808 15:41:59.044329 25661 tablet_service.cc:1467] Processing CreateTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b (DEFAULT_TABLE table=test-table [id=46e12349d88e4c7ca9e862766f47f65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.045871 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d422cafdbc3402d9e0d468bc768aa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.046355 25813 tablet_service.cc:1467] Processing CreateTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b (DEFAULT_TABLE table=test-table [id=46e12349d88e4c7ca9e862766f47f65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.047768 25813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d422cafdbc3402d9e0d468bc768aa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.047797 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.028219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:41325 (local address 127.0.0.1:41426)
0808 15:41:59.038633 (+ 10414us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.038652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.038762 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.040445 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.040450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.040487 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.040897 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.040908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.042695 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.042699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.044775 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.044839 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.047010 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.047017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.047195 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.047228 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.047334 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.047469 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10252,"thread_start_us":10622,"threads_started":1}
I20260808 15:41:59.048076 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.028535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.23.130.1:41325)
0808 15:41:59.039427 (+ 10892us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.039437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.039466 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:41:59.039565 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.039573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.039845 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.040059 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.041065 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.042497 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.045020 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.046531 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.046671 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.046856 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.047402 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.047571 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10700,"thread_start_us":10570,"threads_started":1}
I20260808 15:41:59.051602 25504 tablet_service.cc:1467] Processing CreateTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b (DEFAULT_TABLE table=test-table [id=46e12349d88e4c7ca9e862766f47f65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.052933 25504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d422cafdbc3402d9e0d468bc768aa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.059013 25906 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Bootstrap starting.
I20260808 15:41:59.062076 25906 tablet_bootstrap.cc:654] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.062996 25907 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Bootstrap starting.
I20260808 15:41:59.063526 25906 log.cc:824] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:59.066660 25907 tablet_bootstrap.cc:654] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.066815 25906 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: No bootstrap required, opened a new log
I20260808 15:41:59.067149 25906 ts_tablet_manager.cc:1397] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260808 15:41:59.068002 25909 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Bootstrap starting.
I20260808 15:41:59.068244 25907 log.cc:824] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:59.071211 25909 tablet_bootstrap.cc:654] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.071417 25907 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: No bootstrap required, opened a new log
I20260808 15:41:59.071692 25907 ts_tablet_manager.cc:1397] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:41:59.072293 25909 log.cc:824] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:41:59.074785 25909 tablet_bootstrap.cc:492] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: No bootstrap required, opened a new log
I20260808 15:41:59.075028 25909 ts_tablet_manager.cc:1397] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260808 15:41:59.077817 25906 raft_consensus.cc:348] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.078274 25906 raft_consensus.cc:374] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.078378 25906 raft_consensus.cc:729] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59545169461b491eb0c381a55eda42a8, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.079025 25906 consensus_queue.cc:260] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.080132 25737 heartbeater.cc:503] Master 127.23.130.62:33007 was elected leader, sending a full tablet report...
I20260808 15:41:59.080739 25906 ts_tablet_manager.cc:1428] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260808 15:41:59.082057 25907 raft_consensus.cc:348] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.082504 25907 raft_consensus.cc:374] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.082620 25907 raft_consensus.cc:729] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfbe2edf4b34ca985796186645243b2, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.083228 25907 consensus_queue.cc:260] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.084594 25887 heartbeater.cc:503] Master 127.23.130.62:33007 was elected leader, sending a full tablet report...
I20260808 15:41:59.085150 25907 ts_tablet_manager.cc:1428] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 15:41:59.085657 25909 raft_consensus.cc:348] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.086081 25909 raft_consensus.cc:374] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.086194 25909 raft_consensus.cc:729] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cc5de5774c4b7fb05511c18638af3e, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.086826 25909 consensus_queue.cc:260] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.088656 25909 ts_tablet_manager.cc:1428] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260808 15:41:59.109045 25580 tablet.cc:2369] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:41:59.162678 25738 tablet.cc:2369] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:41:59.183867 25913 raft_consensus.cc:482] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:41:59.184249 25913 raft_consensus.cc:504] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.186218 25913 leader_election.cc:288] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939), a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325)
I20260808 15:41:59.193410 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.186879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44939 (local address 127.23.130.3:48773)
0808 15:41:59.187066 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.187088 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.187216 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.187742 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.187747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.187774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.188024 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.188037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.189549 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.189557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.191089 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.191103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.192663 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.192670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.192858 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.192894 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.192996 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.193135 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:41:59.193879 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.186619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:48773 (local address 127.23.130.2:44939)
0808 15:41:59.186857 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.186863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.187170 (+ 307us) server_negotiation.cc:407] Connection header received
0808 15:41:59.187399 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.187413 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.187489 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.187568 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.188279 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.189413 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.191267 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.192500 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.192656 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.192730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.193039 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.193169 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:41:59.194602 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.187650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59381 (local address 127.23.130.1:41325)
0808 15:41:59.188320 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.188326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.188522 (+ 196us) server_negotiation.cc:407] Connection header received
0808 15:41:59.188658 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.188664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.188745 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.188835 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.189454 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.190547 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.192379 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.193534 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.193700 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.193791 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.194212 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.194342 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506}
I20260808 15:41:59.194672 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.187297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:41325 (local address 127.23.130.3:59381)
0808 15:41:59.188394 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.188416 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.188546 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.188875 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.188880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.188905 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.189256 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.189267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.190701 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.190706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.192211 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.192223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.193831 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.193838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.194000 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.194047 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.194179 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.194300 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":745,"threads_started":1}
I20260808 15:41:59.194881 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d422cafdbc3402d9e0d468bc768aa1b" candidate_uuid: "ddfbe2edf4b34ca985796186645243b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59545169461b491eb0c381a55eda42a8" is_pre_election: true
W20260808 15:41:59.195276 25888 tablet.cc:2369] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:41:59.195474 25683 raft_consensus.cc:2454] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfbe2edf4b34ca985796186645243b2 in term 0.
I20260808 15:41:59.195771 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d422cafdbc3402d9e0d468bc768aa1b" candidate_uuid: "ddfbe2edf4b34ca985796186645243b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cc5de5774c4b7fb05511c18638af3e" is_pre_election: true
I20260808 15:41:59.196184 25761 leader_election.cc:302] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59545169461b491eb0c381a55eda42a8, ddfbe2edf4b34ca985796186645243b2; no voters:
I20260808 15:41:59.196310 25530 raft_consensus.cc:2454] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfbe2edf4b34ca985796186645243b2 in term 0.
I20260808 15:41:59.196588 25913 raft_consensus.cc:2781] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:41:59.196715 25913 raft_consensus.cc:482] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:41:59.196805 25913 raft_consensus.cc:3037] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.199985 25913 raft_consensus.cc:504] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.200646 25913 leader_election.cc:288] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [CANDIDATE]: Term 1 election: Requested vote from peers 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939), a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325)
I20260808 15:41:59.201107 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d422cafdbc3402d9e0d468bc768aa1b" candidate_uuid: "ddfbe2edf4b34ca985796186645243b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59545169461b491eb0c381a55eda42a8"
I20260808 15:41:59.201220 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d422cafdbc3402d9e0d468bc768aa1b" candidate_uuid: "ddfbe2edf4b34ca985796186645243b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cc5de5774c4b7fb05511c18638af3e"
I20260808 15:41:59.201340 25683 raft_consensus.cc:3037] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.201412 25530 raft_consensus.cc:3037] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.204717 25530 raft_consensus.cc:2454] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfbe2edf4b34ca985796186645243b2 in term 1.
I20260808 15:41:59.204717 25683 raft_consensus.cc:2454] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfbe2edf4b34ca985796186645243b2 in term 1.
I20260808 15:41:59.205116 25761 leader_election.cc:302] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59545169461b491eb0c381a55eda42a8, ddfbe2edf4b34ca985796186645243b2; no voters:
I20260808 15:41:59.205577 25913 raft_consensus.cc:2781] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:59.206102 25913 raft_consensus.cc:686] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 1 LEADER]: Becoming Leader. State: Replica: ddfbe2edf4b34ca985796186645243b2, State: Running, Role: LEADER
I20260808 15:41:59.206661 25913 consensus_queue.cc:237] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } }
I20260808 15:41:59.211114 25349 catalog_manager.cc:5495] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfbe2edf4b34ca985796186645243b2 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "ddfbe2edf4b34ca985796186645243b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } health_report { overall_health: UNKNOWN } } }
I20260808 15:41:59.236174 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.230680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.23.130.3:36691)
0808 15:41:59.230937 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.230944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.231031 (+ 87us) server_negotiation.cc:407] Connection header received
0808 15:41:59.231314 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.231318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.231406 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.231530 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:41:59.231915 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.232621 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.233924 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.234324 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.234485 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.234950 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:41:59.235451 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:41:59.235631 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.235741 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.235807 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:41:59.244366 25530 raft_consensus.cc:1261] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Refusing update from remote peer ddfbe2edf4b34ca985796186645243b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:41:59.244387 25683 raft_consensus.cc:1261] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Refusing update from remote peer ddfbe2edf4b34ca985796186645243b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:41:59.245455 25917 consensus_queue.cc:1035] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:41:59.245863 25913 consensus_queue.cc:1035] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:41:59.258106 25920 mvcc.cc:205] Tried to move back new op lower bound from 7316290434014605312 to 7316290433874071552. Current Snapshot: MvccSnapshot[applied={T|T < 7316290434014605312}]
I20260808 15:41:59.263192 25921 mvcc.cc:205] Tried to move back new op lower bound from 7316290434014605312 to 7316290433874071552. Current Snapshot: MvccSnapshot[applied={T|T < 7316290434014605312}]
I20260808 15:41:59.277583 25349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:41:59.277983 25349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:41:59.282073 25349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Sending DeleteTablet for 3 replicas of tablet 8d422cafdbc3402d9e0d468bc768aa1b
I20260808 15:41:59.283321 25813 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:41:59 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:42752
I20260808 15:41:59.283571 25504 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:41:59 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:41426
I20260808 15:41:59.284104 25661 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d422cafdbc3402d9e0d468bc768aa1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:41:59 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:39882
I20260808 15:41:59.285610 25349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:41:59.286716 25349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:41:59.289673 25935 tablet_replica.cc:330] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: stopping tablet replica
I20260808 15:41:59.290378 25935 raft_consensus.cc:2229] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:41:59.291117 25935 raft_consensus.cc:2258] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:41:59.292065 25934 tablet_replica.cc:330] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: stopping tablet replica
I20260808 15:41:59.292567 25934 raft_consensus.cc:2229] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:41:59.293097 25934 raft_consensus.cc:2258] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:41:59.293565 25935 ts_tablet_manager.cc:1905] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:41:59.295202 25812 tablet_service.cc:1467] Processing CreateTablet for tablet 7d25d889791541d6879de9268635a565 (DEFAULT_TABLE table=test-table [id=cc4f54282ef94485a96a40f67837d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.295849 25812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d25d889791541d6879de9268635a565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.302901 25504 tablet_service.cc:1467] Processing CreateTablet for tablet 7d25d889791541d6879de9268635a565 (DEFAULT_TABLE table=test-table [id=cc4f54282ef94485a96a40f67837d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.303125 25661 tablet_service.cc:1467] Processing CreateTablet for tablet 7d25d889791541d6879de9268635a565 (DEFAULT_TABLE table=test-table [id=cc4f54282ef94485a96a40f67837d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:41:59.303428 25504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d25d889791541d6879de9268635a565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.303583 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d25d889791541d6879de9268635a565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.308070 25933 tablet_replica.cc:330] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: stopping tablet replica
I20260808 15:41:59.308604 25933 raft_consensus.cc:2229] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:41:59.309451 25933 raft_consensus.cc:2258] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:41:59.309947 25906 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Bootstrap starting.
I20260808 15:41:59.310485 25909 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Bootstrap starting.
I20260808 15:41:59.311318 25907 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Bootstrap starting.
I20260808 15:41:59.311674 25906 tablet_bootstrap.cc:654] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.312108 25909 tablet_bootstrap.cc:654] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.312325 25935 ts_tablet_manager.cc:1918] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:41:59.313489 25907 tablet_bootstrap.cc:654] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.313746 25935 log.cc:1196] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/8d422cafdbc3402d9e0d468bc768aa1b
I20260808 15:41:59.314566 25935 ts_tablet_manager.cc:1939] T 8d422cafdbc3402d9e0d468bc768aa1b P 59545169461b491eb0c381a55eda42a8: Deleting consensus metadata
I20260808 15:41:59.314854 25934 ts_tablet_manager.cc:1905] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:41:59.319972 25933 ts_tablet_manager.cc:1905] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:41:59.321612 25333 catalog_manager.cc:4841] TS 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939): tablet 8d422cafdbc3402d9e0d468bc768aa1b (table test-table [id=46e12349d88e4c7ca9e862766f47f65c]) successfully deleted
I20260808 15:41:59.327739 25907 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: No bootstrap required, opened a new log
I20260808 15:41:59.327924 25907 ts_tablet_manager.cc:1397] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260808 15:41:59.328377 25906 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: No bootstrap required, opened a new log
I20260808 15:41:59.328531 25906 ts_tablet_manager.cc:1397] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260808 15:41:59.328922 25907 raft_consensus.cc:348] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.329223 25907 raft_consensus.cc:374] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.329360 25907 raft_consensus.cc:729] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfbe2edf4b34ca985796186645243b2, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.329402 25906 raft_consensus.cc:348] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.329610 25906 raft_consensus.cc:374] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.329706 25906 raft_consensus.cc:729] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59545169461b491eb0c381a55eda42a8, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.329670 25907 consensus_queue.cc:260] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.329943 25906 consensus_queue.cc:260] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.332070 25907 ts_tablet_manager.cc:1428] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 15:41:59.332528 25934 ts_tablet_manager.cc:1918] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:41:59.332849 25934 log.cc:1196] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/8d422cafdbc3402d9e0d468bc768aa1b
I20260808 15:41:59.333673 25934 ts_tablet_manager.cc:1939] T 8d422cafdbc3402d9e0d468bc768aa1b P a0cc5de5774c4b7fb05511c18638af3e: Deleting consensus metadata
I20260808 15:41:59.335072 25906 ts_tablet_manager.cc:1428] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 15:41:59.336048 25333 catalog_manager.cc:4841] TS a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325): tablet 8d422cafdbc3402d9e0d468bc768aa1b (table test-table [id=46e12349d88e4c7ca9e862766f47f65c]) successfully deleted
I20260808 15:41:59.336787 25909 tablet_bootstrap.cc:492] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: No bootstrap required, opened a new log
I20260808 15:41:59.337024 25909 ts_tablet_manager.cc:1397] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.015s
I20260808 15:41:59.337852 25909 raft_consensus.cc:348] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.338080 25909 raft_consensus.cc:374] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.338176 25909 raft_consensus.cc:729] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cc5de5774c4b7fb05511c18638af3e, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.338517 25909 consensus_queue.cc:260] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.338361 25933 ts_tablet_manager.cc:1918] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:41:59.338987 25933 log.cc:1196] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/8d422cafdbc3402d9e0d468bc768aa1b
I20260808 15:41:59.339093 25909 ts_tablet_manager.cc:1428] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 15:41:59.339596 25933 ts_tablet_manager.cc:1939] T 8d422cafdbc3402d9e0d468bc768aa1b P ddfbe2edf4b34ca985796186645243b2: Deleting consensus metadata
I20260808 15:41:59.341423 25331 catalog_manager.cc:4841] TS ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691): tablet 8d422cafdbc3402d9e0d468bc768aa1b (table test-table [id=46e12349d88e4c7ca9e862766f47f65c]) successfully deleted
I20260808 15:41:59.617224 25912 raft_consensus.cc:482] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:41:59.617494 25912 raft_consensus.cc:504] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.619321 25912 leader_election.cc:288] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325), ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691)
I20260808 15:41:59.628582 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.619793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:53425 (local address 127.23.130.1:41325)
0808 15:41:59.619997 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.620005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.621337 (+ 1332us) server_negotiation.cc:407] Connection header received
0808 15:41:59.621634 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.621639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.621699 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.621802 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.622323 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.623401 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.626088 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.627351 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.627516 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.627582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.628157 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.628293 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:41:59.628607 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.620313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:41247 (local address 127.23.130.3:36691)
0808 15:41:59.620585 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.620594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.621597 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 15:41:59.621863 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.621869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.621965 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.622036 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.622877 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.623865 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.625918 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.627319 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.627506 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.627585 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.628119 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.628289 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:41:59.628733 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.619854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36691 (local address 127.23.130.2:41247)
0808 15:41:59.621481 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.621512 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.621701 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.622238 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.622244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.622269 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.622703 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.622717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.624013 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.624018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.625774 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.625789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.627565 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.627617 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.627845 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.627955 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.628060 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.628244 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":33,"thread_start_us":934,"threads_started":1}
I20260808 15:41:59.629849 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.619772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:41325 (local address 127.23.130.2:53425)
0808 15:41:59.621204 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.621229 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.621466 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.621823 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.621827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.621853 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.622144 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.622152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.623555 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.623562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.625916 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.625933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.627752 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.627759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.627950 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.627975 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.629482 (+ 1507us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.629578 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":34,"thread_start_us":1365,"threads_started":1}
I20260808 15:41:59.629926 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d25d889791541d6879de9268635a565" candidate_uuid: "59545169461b491eb0c381a55eda42a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfbe2edf4b34ca985796186645243b2" is_pre_election: true
I20260808 15:41:59.630540 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d25d889791541d6879de9268635a565" candidate_uuid: "59545169461b491eb0c381a55eda42a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cc5de5774c4b7fb05511c18638af3e" is_pre_election: true
I20260808 15:41:59.630600 25835 raft_consensus.cc:2454] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59545169461b491eb0c381a55eda42a8 in term 0.
I20260808 15:41:59.630738 25530 raft_consensus.cc:2454] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59545169461b491eb0c381a55eda42a8 in term 0.
I20260808 15:41:59.631273 25603 leader_election.cc:302] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 59545169461b491eb0c381a55eda42a8, a0cc5de5774c4b7fb05511c18638af3e, ddfbe2edf4b34ca985796186645243b2; no voters:
I20260808 15:41:59.631791 25912 raft_consensus.cc:2781] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:41:59.631924 25912 raft_consensus.cc:482] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:41:59.632032 25912 raft_consensus.cc:3037] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.634264 25912 raft_consensus.cc:504] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.634987 25912 leader_election.cc:288] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [CANDIDATE]: Term 1 election: Requested vote from peers a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325), ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691)
I20260808 15:41:59.635434 25530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d25d889791541d6879de9268635a565" candidate_uuid: "59545169461b491eb0c381a55eda42a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cc5de5774c4b7fb05511c18638af3e"
I20260808 15:41:59.635602 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d25d889791541d6879de9268635a565" candidate_uuid: "59545169461b491eb0c381a55eda42a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfbe2edf4b34ca985796186645243b2"
I20260808 15:41:59.635704 25530 raft_consensus.cc:3037] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.635818 25835 raft_consensus.cc:3037] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.638068 25530 raft_consensus.cc:2454] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59545169461b491eb0c381a55eda42a8 in term 1.
I20260808 15:41:59.638520 25603 leader_election.cc:302] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59545169461b491eb0c381a55eda42a8, a0cc5de5774c4b7fb05511c18638af3e; no voters:
I20260808 15:41:59.638808 25912 raft_consensus.cc:2781] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:59.639336 25912 raft_consensus.cc:686] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 1 LEADER]: Becoming Leader. State: Replica: 59545169461b491eb0c381a55eda42a8, State: Running, Role: LEADER
I20260808 15:41:59.639480 25835 raft_consensus.cc:2454] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59545169461b491eb0c381a55eda42a8 in term 1.
I20260808 15:41:59.639981 25912 consensus_queue.cc:237] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.643590 25349 catalog_manager.cc:5495] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59545169461b491eb0c381a55eda42a8 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "59545169461b491eb0c381a55eda42a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:41:59 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203717, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:41:59.712433 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.697030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33007 (local address 127.0.0.1:44352)
0808 15:41:59.697719 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.697740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.697896 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.698363 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.698368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.699266 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:41:59.699542 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.699557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.700681 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.700687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.702216 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.702231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.703020 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.703026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.703227 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.704001 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:41:59.704024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:41:59.706630 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:41:59.709832 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:59.709844 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:59.709852 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:59.710275 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:59.710681 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:41:59.710686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:41:59.710690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:41:59.710833 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:41:59.711340 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:41:59.711348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:41:59.711538 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.711714 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.712014 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":350,"threads_started":1}
I20260808 15:41:59.712527 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.697002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.23.130.62:33007)
0808 15:41:59.698062 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.698066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.698087 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:41:59.698130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.698133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.698190 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.698280 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:41:59.699694 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.700547 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.702349 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.702856 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.702951 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.706860 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:41:59.706865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:41:59.706868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:41:59.706895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:41:59.709676 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:59.710398 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:59.710428 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:59.710431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:59.710566 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:41:59.710943 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:41:59.710950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:41:59.710953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:41:59.711201 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:41:59.711381 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.711802 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.712146 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":956,"thread_start_us":811,"threads_started":1}
I20260808 15:41:59.718034 25349 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:44352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:41:59.719228 25349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:41:59.726274 25504 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f1d49f0d488ebbbf881935a622f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e18102c6144bed87178638da96fb3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:41:59.726671 25661 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f1d49f0d488ebbbf881935a622f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e18102c6144bed87178638da96fb3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:41:59.726895 25504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f1d49f0d488ebbbf881935a622f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.726773 25812 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f1d49f0d488ebbbf881935a622f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e18102c6144bed87178638da96fb3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:41:59.727192 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f1d49f0d488ebbbf881935a622f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.727295 25812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f1d49f0d488ebbbf881935a622f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:41:59.733536 25907 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: Bootstrap starting.
I20260808 15:41:59.734104 25909 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: Bootstrap starting.
I20260808 15:41:59.735335 25907 tablet_bootstrap.cc:654] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.735881 25909 tablet_bootstrap.cc:654] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.737502 25907 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: No bootstrap required, opened a new log
I20260808 15:41:59.737596 25906 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: Bootstrap starting.
I20260808 15:41:59.737679 25907 ts_tablet_manager.cc:1397] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:41:59.738055 25909 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: No bootstrap required, opened a new log
I20260808 15:41:59.738232 25909 ts_tablet_manager.cc:1397] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:41:59.738514 25907 raft_consensus.cc:348] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.738786 25907 raft_consensus.cc:374] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.738870 25907 raft_consensus.cc:729] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfbe2edf4b34ca985796186645243b2, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.739046 25909 raft_consensus.cc:348] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.739279 25909 raft_consensus.cc:374] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.739125 25907 consensus_queue.cc:260] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.739363 25906 tablet_bootstrap.cc:654] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: Neither blocks nor log segments found. Creating new log.
I20260808 15:41:59.739360 25909 raft_consensus.cc:729] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cc5de5774c4b7fb05511c18638af3e, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.739645 25917 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:41:59.739701 25909 consensus_queue.cc:260] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.739948 25917 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:41:59.740132 25915 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:41:59.740478 25915 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:41:59.741374 25909 ts_tablet_manager.cc:1428] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 15:41:59.742813 25907 ts_tablet_manager.cc:1428] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 15:41:59.743386 25906 tablet_bootstrap.cc:492] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: No bootstrap required, opened a new log
I20260808 15:41:59.743577 25906 ts_tablet_manager.cc:1397] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 15:41:59.744182 25906 raft_consensus.cc:348] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.744360 25906 raft_consensus.cc:374] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:41:59.744453 25906 raft_consensus.cc:729] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59545169461b491eb0c381a55eda42a8, State: Initialized, Role: FOLLOWER
I20260808 15:41:59.744762 25906 consensus_queue.cc:260] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.745247 25912 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:41:59.745383 25906 ts_tablet_manager.cc:1428] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260808 15:41:59.745545 25912 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:41:59.975627 25915 raft_consensus.cc:482] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:41:59.975828 25915 raft_consensus.cc:504] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.977201 25915 leader_election.cc:288] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939), ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691)
I20260808 15:41:59.984045 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.977622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:55681 (local address 127.23.130.3:36691)
0808 15:41:59.977828 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.977838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.978500 (+ 662us) server_negotiation.cc:407] Connection header received
0808 15:41:59.978843 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.978851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.978928 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.978981 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.979646 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.980319 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.982170 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.983011 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.983201 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.983283 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.983633 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.983774 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:41:59.984184 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.977546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36691 (local address 127.23.130.1:55681)
0808 15:41:59.978324 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.978349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.978655 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.979138 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.979141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.979160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.979446 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.979453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.980467 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.980474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.981998 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.982013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.983220 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.983226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.983446 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.983478 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.983596 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.983749 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":80,"thread_start_us":450,"threads_started":1}
I20260808 15:41:59.984205 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.977667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44939 (local address 127.23.130.1:48291)
0808 15:41:59.978761 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0808 15:41:59.978788 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:41:59.978906 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:41:59.979285 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:41:59.979290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:41:59.979318 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:41:59.979726 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.979735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.980770 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.980777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.982130 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:41:59.982142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.983396 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.983402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:41:59.983562 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.983584 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:41:59.983703 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:41:59.983829 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":808,"threads_started":1}
I20260808 15:41:59.984234 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:41:59.977622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:48291 (local address 127.23.130.2:44939)
0808 15:41:59.979006 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0808 15:41:59.979012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:41:59.979034 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:41:59.979092 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:41:59.979096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:41:59.979149 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:41:59.979265 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:41:59.979910 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.980631 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.982292 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:41:59.983136 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:41:59.983323 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:41:59.983402 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:41:59.983747 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 15:41:59.983892 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1121,"threads_started":1}
I20260808 15:41:59.984932 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f1d49f0d488ebbbf881935a622f1" candidate_uuid: "a0cc5de5774c4b7fb05511c18638af3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfbe2edf4b34ca985796186645243b2" is_pre_election: true
I20260808 15:41:59.984977 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f1d49f0d488ebbbf881935a622f1" candidate_uuid: "a0cc5de5774c4b7fb05511c18638af3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59545169461b491eb0c381a55eda42a8" is_pre_election: true
I20260808 15:41:59.985208 25835 raft_consensus.cc:2454] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0cc5de5774c4b7fb05511c18638af3e in term 0.
I20260808 15:41:59.985203 25683 raft_consensus.cc:2454] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0cc5de5774c4b7fb05511c18638af3e in term 0.
I20260808 15:41:59.985708 25450 leader_election.cc:302] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59545169461b491eb0c381a55eda42a8, a0cc5de5774c4b7fb05511c18638af3e; no voters:
I20260808 15:41:59.986109 25915 raft_consensus.cc:2781] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:41:59.986249 25915 raft_consensus.cc:482] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:41:59.986366 25915 raft_consensus.cc:3037] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.988691 25915 raft_consensus.cc:504] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.989240 25915 leader_election.cc:288] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [CANDIDATE]: Term 1 election: Requested vote from peers 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939), ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691)
I20260808 15:41:59.989755 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f1d49f0d488ebbbf881935a622f1" candidate_uuid: "a0cc5de5774c4b7fb05511c18638af3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfbe2edf4b34ca985796186645243b2"
I20260808 15:41:59.989756 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f1d49f0d488ebbbf881935a622f1" candidate_uuid: "a0cc5de5774c4b7fb05511c18638af3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59545169461b491eb0c381a55eda42a8"
I20260808 15:41:59.989970 25683 raft_consensus.cc:3037] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.989985 25835 raft_consensus.cc:3037] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:41:59.991988 25835 raft_consensus.cc:2454] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0cc5de5774c4b7fb05511c18638af3e in term 1.
I20260808 15:41:59.992036 25683 raft_consensus.cc:2454] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0cc5de5774c4b7fb05511c18638af3e in term 1.
I20260808 15:41:59.992465 25447 leader_election.cc:302] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0cc5de5774c4b7fb05511c18638af3e, ddfbe2edf4b34ca985796186645243b2; no voters:
I20260808 15:41:59.992800 25915 raft_consensus.cc:2781] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:41:59.993206 25915 raft_consensus.cc:686] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [term 1 LEADER]: Becoming Leader. State: Replica: a0cc5de5774c4b7fb05511c18638af3e, State: Running, Role: LEADER
I20260808 15:41:59.993822 25915 consensus_queue.cc:237] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } }
I20260808 15:41:59.995842 25957 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: TxnStatusTablet state changed. Reason: New leader a0cc5de5774c4b7fb05511c18638af3e. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:41:59.996325 25957 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:41:59.997217 25959 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:41:59.997694 25959 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:41:59.997834 25347 catalog_manager.cc:5495] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1). New cstate: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:00.036857 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.030600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:41325 (local address 127.0.0.1:41436)
0808 15:42:00.030925 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:00.030942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:00.031121 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:00.031681 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:00.031686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:00.031698 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:00.031931 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.031940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.033138 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.033146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.034177 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.034189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.035056 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.035062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.035260 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.035318 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.036229 (+ 911us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.036251 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:42:00.036382 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 15:42:00.036546 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 15:42:00.036859 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.030595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.23.130.1:41325)
0808 15:42:00.030785 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.030791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.031026 (+ 235us) server_negotiation.cc:407] Connection header received
0808 15:42:00.031344 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.031349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.031508 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.031657 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:00.032077 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.033000 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.034349 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.034879 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.035022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.035524 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.036107 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.036279 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.036422 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.036513 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:42:00.042263 25683 raft_consensus.cc:1261] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Refusing update from remote peer a0cc5de5774c4b7fb05511c18638af3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:00.042490 25835 raft_consensus.cc:1261] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Refusing update from remote peer a0cc5de5774c4b7fb05511c18638af3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:00.043124 25957 consensus_queue.cc:1035] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:00.043530 25915 consensus_queue.cc:1035] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:00.049863 25912 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: TxnStatusTablet state changed. Reason: New leader a0cc5de5774c4b7fb05511c18638af3e. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.050240 25912 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:00.052006 25941 consensus_queue.cc:1035] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:00.059868 25917 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: TxnStatusTablet state changed. Reason: New leader a0cc5de5774c4b7fb05511c18638af3e. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.060101 25962 mvcc.cc:205] Tried to move back new op lower bound from 7316290437285412864 to 7316290437098668032. Current Snapshot: MvccSnapshot[applied={T|T < 7316290437285412864}]
I20260808 15:42:00.060217 25917 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:00.065788 25941 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.066208 25941 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P 59545169461b491eb0c381a55eda42a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:00.066747 25917 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.067016 25917 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P ddfbe2edf4b34ca985796186645243b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:00.069196 25915 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.069805 25915 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:00.070181 25915 tablet_replica.cc:439] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0cc5de5774c4b7fb05511c18638af3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0cc5de5774c4b7fb05511c18638af3e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 41325 } } peers { permanent_uuid: "59545169461b491eb0c381a55eda42a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44939 } } peers { permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 } } }
I20260808 15:42:00.070550 25915 tablet_replica.cc:442] T 12b2f1d49f0d488ebbbf881935a622f1 P a0cc5de5774c4b7fb05511c18638af3e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:00.078269 25912 consensus_queue.cc:1035] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfbe2edf4b34ca985796186645243b2" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:00.094601 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.088008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.23.130.2:44939)
0808 15:42:00.088547 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.088553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.088818 (+ 265us) server_negotiation.cc:407] Connection header received
0808 15:42:00.089091 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.089096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.089197 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.089399 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:00.089907 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.090755 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.092231 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.092665 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.092813 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.093267 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.093828 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.094009 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.094123 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.094212 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260808 15:42:00.113186 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.106212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:52927 (local address 127.23.130.1:41325)
0808 15:42:00.106466 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.106474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.107088 (+ 614us) server_negotiation.cc:407] Connection header received
0808 15:42:00.107281 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.107285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.107352 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.107432 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:00.108004 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.108729 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.111048 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.111783 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.111902 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.112503 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.112629 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.112706 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.112873 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.112943 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:42:00.113412 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.106162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:41325 (local address 127.23.130.2:52927)
0808 15:42:00.106970 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:00.106991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:00.107149 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:00.107484 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:00.107490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:00.107503 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:00.107846 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.107857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.108874 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.108879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.110845 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.110885 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.111996 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.112003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.112191 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.112268 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.112744 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.112756 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 15:42:00.112845 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 15:42:00.113002 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":472,"threads_started":1}
I20260808 15:42:00.161923 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.149355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:44789 (local address 127.23.130.2:44939)
0808 15:42:00.149587 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.149597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.155690 (+ 6093us) server_negotiation.cc:407] Connection header received
0808 15:42:00.155984 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.155992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.156081 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.156187 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:00.156852 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.157556 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.159918 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.160349 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.160454 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.161007 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.161171 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.161261 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.161521 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.161601 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:42:00.162273 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.148885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44939 (local address 127.23.130.1:44789)
0808 15:42:00.155550 (+ 6665us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:00.155572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:00.155789 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:00.156228 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:00.156234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:00.156247 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:00.156677 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.156691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.157733 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.157739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.159752 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:00.159768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.160546 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.160553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:00.160748 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.160823 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:00.161312 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:00.161359 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:42:00.161490 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 15:42:00.161635 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6417,"thread_start_us":6277,"threads_started":1}
I20260808 15:42:00.204879 25347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:00.205156 25347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:00.208979 25347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4329ae733cd6498fae3e2aa214d730bc: Sending DeleteTablet for 3 replicas of tablet 7d25d889791541d6879de9268635a565
I20260808 15:42:00.210223 25812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d25d889791541d6879de9268635a565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:00 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:42752
I20260808 15:42:00.210970 25504 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d25d889791541d6879de9268635a565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:00 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:41426
I20260808 15:42:00.211594 25661 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d25d889791541d6879de9268635a565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:00 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:39882
I20260808 15:42:00.212349 25985 tablet_replica.cc:330] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: stopping tablet replica
I20260808 15:42:00.212683 25985 raft_consensus.cc:2229] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:00.213078 25985 raft_consensus.cc:2258] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:00.214643 25985 ts_tablet_manager.cc:1905] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:00.223692 25985 ts_tablet_manager.cc:1918] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:00.224085 25985 log.cc:1196] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/7d25d889791541d6879de9268635a565
I20260808 15:42:00.224900 25985 ts_tablet_manager.cc:1939] T 7d25d889791541d6879de9268635a565 P ddfbe2edf4b34ca985796186645243b2: Deleting consensus metadata
I20260808 15:42:00.226150 25331 catalog_manager.cc:4841] TS ddfbe2edf4b34ca985796186645243b2 (127.23.130.3:36691): tablet 7d25d889791541d6879de9268635a565 (table test-table [id=cc4f54282ef94485a96a40f67837d9df]) successfully deleted
I20260808 15:42:00.232043 25986 tablet_replica.cc:330] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: stopping tablet replica
I20260808 15:42:00.232259 25986 raft_consensus.cc:2229] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:00.232573 25986 raft_consensus.cc:2258] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:00.234768 25986 ts_tablet_manager.cc:1905] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:00.238626 25987 tablet_replica.cc:330] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: stopping tablet replica
I20260808 15:42:00.238855 25987 raft_consensus.cc:2229] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:00.239504 25987 raft_consensus.cc:2258] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:00.240715 25987 ts_tablet_manager.cc:1905] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:00.249931 25986 ts_tablet_manager.cc:1918] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:00.250089 25986 log.cc:1196] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/7d25d889791541d6879de9268635a565
I20260808 15:42:00.250594 25986 ts_tablet_manager.cc:1939] T 7d25d889791541d6879de9268635a565 P a0cc5de5774c4b7fb05511c18638af3e: Deleting consensus metadata
I20260808 15:42:00.251721 25333 catalog_manager.cc:4841] TS a0cc5de5774c4b7fb05511c18638af3e (127.23.130.1:41325): tablet 7d25d889791541d6879de9268635a565 (table test-table [id=cc4f54282ef94485a96a40f67837d9df]) successfully deleted
I20260808 15:42:00.253728 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.227215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.23.130.1:41325)
0808 15:42:00.227460 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.227469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.231058 (+ 3589us) server_negotiation.cc:407] Connection header received
0808 15:42:00.231688 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.231698 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.231793 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.231870 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:00.233933 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.235136 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.237529 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.238200 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.238304 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.244236 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:00.244253 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:00.244272 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:00.244333 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:00.249129 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.250456 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.250465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.250469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.250702 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.251200 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.251206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.251209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.252140 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:00.252433 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.253100 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.253335 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 15:42:00.255424 25574 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41444
I20260808 15:42:00.255633 25987 ts_tablet_manager.cc:1918] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:00.255784 25987 log.cc:1196] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/7d25d889791541d6879de9268635a565
I20260808 15:42:00.256299 25987 ts_tablet_manager.cc:1939] T 7d25d889791541d6879de9268635a565 P 59545169461b491eb0c381a55eda42a8: Deleting consensus metadata
I20260808 15:42:00.257705 25333 catalog_manager.cc:4841] TS 59545169461b491eb0c381a55eda42a8 (127.23.130.2:44939): tablet 7d25d889791541d6879de9268635a565 (table test-table [id=cc4f54282ef94485a96a40f67837d9df]) successfully deleted
I20260808 15:42:00.292595 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.264009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.23.130.62:33007)
0808 15:42:00.275130 (+ 11121us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.275137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.275156 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:00.275222 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.275227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.275319 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.275444 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:00.277306 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.278862 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.280796 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.281545 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.281695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.285069 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:00.285078 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:00.285084 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:00.285165 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:00.289292 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.290114 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.290121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.290125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.290292 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.290764 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.290770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.290774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.291192 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:00.291482 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.291897 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.292171 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10984,"thread_start_us":10876,"threads_started":1}
W20260808 15:42:00.293277 25347 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368
I20260808 15:42:00.295958 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25430
I20260808 15:42:00.315701 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.297821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.23.130.1:41325)
0808 15:42:00.298014 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.298019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.298469 (+ 450us) server_negotiation.cc:407] Connection header received
0808 15:42:00.299000 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.299012 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.299089 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.299159 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:00.300889 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.302052 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.303973 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.304763 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.304882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.308088 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:00.308095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:00.308100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:00.308139 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:00.312090 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.312989 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.312996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.312999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.313165 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.313630 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.313636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.313640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.314021 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:00.314270 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.314915 (+ 645us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.315174 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260808 15:42:00.316375 25574 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41458
W20260808 15:42:00.317042 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:00.317245 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 25430
I20260808 15:42:00.341230 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25583
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:00.357610 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.342139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.23.130.2:44939)
0808 15:42:00.342321 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.342329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.342350 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:42:00.342639 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.342644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.342712 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.342838 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:00.343996 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.344880 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.346311 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.346947 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.347092 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.351346 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:00.351360 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:00.351380 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:00.351412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:00.354612 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.355323 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.355341 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.355344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.355496 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.355910 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.355917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.355920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.356355 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:00.356687 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.356975 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.357209 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:42:00.358471 25732 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898
W20260808 15:42:00.359288 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:00.359509 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 25583
I20260808 15:42:00.382021 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25741
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:00.396930 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:00.382846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.23.130.3:36691)
0808 15:42:00.383023 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:00.383028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:00.383045 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:00.383203 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:00.383207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:00.383280 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:00.383367 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:00.384625 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.385319 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.386584 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:00.386999 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:00.387092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:00.391426 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:00.391440 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:00.391470 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:00.391509 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:00.394230 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.394991 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.394998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.395001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.395142 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:00.395584 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:00.395587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:00.395589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:00.395943 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:00.396191 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:00.396484 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:42:00.396661 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 15:42:00.397706 25882 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42766
W20260808 15:42:00.398501 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:00.398662 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 25741
I20260808 15:42:00.423540 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 25312
I20260808 15:42:00.424767 25394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:00.541242 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 25312
2026-08-08T15:42:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5161 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25998](info): set up 2 sockets
Aug 08 15:42:00 dist-test-slave-gp6n krb5kdc[25998](info): commencing operation
krb5kdc: starting...
W20260808 15:42:01.836257 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 15:42:01.987464 24072 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.151s user 0.000s sys 0.007s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 15:42:02.247649 24072 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.260s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 15:42:02.449834 24072 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.202s user 0.001s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/test-user.keytab.
W20260808 15:42:02.860008 24072 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.410s user 0.000s sys 0.006s
Aug 08 15:42:02 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
W20260808 15:42:03.342203 24072 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.130.62: real 0.422s user 0.001s sys 0.011s
I20260808 15:42:03.343369 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:36893
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:03.670802 26012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:03.671165 26012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:03.671350 26012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:03.678534 26012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:03.678682 26012 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:42:03.678782 26012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:03.678916 26012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:03.679013 26012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:03.679103 26012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:03.688810 26012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:36893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:03.690124 26012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:03.691493 26012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:03.703578 26018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:03.703492 26023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:03.703450 26019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:03.704726 26012 server_base.cc:1034] running on GCE node
I20260808 15:42:03.705494 26012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:03.707062 26012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:03.708317 26012 hybrid_clock.cc:648] HybridClock initialized: now 1786203723708231 us; error 62 us; skew 500 ppm
I20260808 15:42:03.708652 26012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:03 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203723, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:03.717092 26012 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:03.718889 26012 webserver.cc:460] Webserver started at http://127.23.130.62:44003/ using document root <none> and password file <none>
I20260808 15:42:03.719708 26012 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:03.719856 26012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:03.720206 26012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:03.723699 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "08662419dc4a47c78582741dd09ad919"
format_stamp: "Formatted at 2026-08-08 15:42:03 on dist-test-slave-gp6n"
server_key: "31581908d2a13775665ff3356ccac154"
server_key_iv: "235f9928c44148a081ebb296c4b3d3f6"
server_key_version: "encryptionkey@0"
I20260808 15:42:03.724572 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "08662419dc4a47c78582741dd09ad919"
format_stamp: "Formatted at 2026-08-08 15:42:03 on dist-test-slave-gp6n"
server_key: "31581908d2a13775665ff3356ccac154"
server_key_iv: "235f9928c44148a081ebb296c4b3d3f6"
server_key_version: "encryptionkey@0"
I20260808 15:42:03.730024 26012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:42:03.733731 26031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:03.735069 26012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:03.735256 26012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "08662419dc4a47c78582741dd09ad919"
format_stamp: "Formatted at 2026-08-08 15:42:03 on dist-test-slave-gp6n"
server_key: "31581908d2a13775665ff3356ccac154"
server_key_iv: "235f9928c44148a081ebb296c4b3d3f6"
server_key_version: "encryptionkey@0"
I20260808 15:42:03.735472 26012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:03.757164 26012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:03.761893 26012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:03.762208 26012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:03.784368 26012 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:36893
I20260808 15:42:03.784391 26098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:36893 every 8 connection(s)
I20260808 15:42:03.785755 26012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:03.789615 26099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:03.794432 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26012
I20260808 15:42:03.794682 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:03.795531 24072 external_mini_cluster.cc:1428] Setting key 1b723322f88b1d5f4c75d91f46e0eb7e
I20260808 15:42:03.801085 26099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Bootstrap starting.
I20260808 15:42:03.804967 26099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:03.806214 26099 log.cc:824] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:03.809744 26099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: No bootstrap required, opened a new log
Aug 08 15:42:03 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:03.817982 26099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } }
I20260808 15:42:03.818503 26099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:03.818631 26099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08662419dc4a47c78582741dd09ad919, State: Initialized, Role: FOLLOWER
I20260808 15:42:03.819303 26099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } }
I20260808 15:42:03.819583 26099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:03.819599 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:03.797644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.23.130.62:36893)
0808 15:42:03.798557 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:03.798569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:03.798617 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:42:03.800254 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:03.800277 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:03.800659 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:03.800976 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:03.802827 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:03.804160 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:03.806008 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:03.806971 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:03.807193 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:03.811761 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:03.811773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:03.811798 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:03.811858 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:03.815277 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:03.815972 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:03.815984 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:03.815991 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:03.816172 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:03.816612 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:03.816617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:03.816621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:03.817165 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:03.817497 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:03.817915 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 15:42:03.818367 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260808 15:42:03.819759 26099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:03.819983 26099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:03.822329 26099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } }
I20260808 15:42:03.822856 26099 leader_election.cc:302] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08662419dc4a47c78582741dd09ad919; no voters:
I20260808 15:42:03.823356 26099 leader_election.cc:288] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:03.823496 26105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:03.824179 26105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [term 1 LEADER]: Becoming Leader. State: Replica: 08662419dc4a47c78582741dd09ad919, State: Running, Role: LEADER
I20260808 15:42:03.824769 26105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } }
I20260808 15:42:03.825368 26099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:03.831861 26106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08662419dc4a47c78582741dd09ad919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } } }
I20260808 15:42:03.831827 26108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08662419dc4a47c78582741dd09ad919. Latest consensus state: current_term: 1 leader_uuid: "08662419dc4a47c78582741dd09ad919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08662419dc4a47c78582741dd09ad919" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 36893 } } }
I20260808 15:42:03.832248 26106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:03.832248 26108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919 [sys.catalog]: This master's current role is: LEADER
W20260808 15:42:03.836687 26120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:42:03.836815 26120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:42:03.836916 26121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:03.839020 26121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:03.847733 26121 catalog_manager.cc:1347] Generated new cluster ID: 287d06c065c143baa6de4674853cdd9c
I20260808 15:42:03.847872 26121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:03.887661 26121 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:03.889259 26121 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:03.896829 26121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Generated new TSK 0
I20260808 15:42:03.897693 26121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:03.958206 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:36893
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:04.304454 26127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:04.304744 26127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:04.304915 26127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:04.312206 26127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:04.312312 26127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:04.312495 26127 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:42:04.312582 26127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:04.312721 26127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:04.323045 26127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:04.324337 26127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:04.325757 26127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:04.339255 26135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:04.339337 26133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:04.339340 26132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:04.340365 26127 server_base.cc:1034] running on GCE node
I20260808 15:42:04.340934 26127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:04.341794 26127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:04.342995 26127 hybrid_clock.cc:648] HybridClock initialized: now 1786203724342952 us; error 32 us; skew 500 ppm
I20260808 15:42:04.343264 26127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:04 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:04.346555 26127 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:04.347922 26127 webserver.cc:460] Webserver started at http://127.23.130.1:34771/ using document root <none> and password file <none>
I20260808 15:42:04.348562 26127 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:04.348665 26127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:04.348912 26127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:04.351802 26127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "540cf6e419e341af90d64164d5e2f16d"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "b5dfe22fcc0078b86a61fb8a9fd28415"
server_key_iv: "f5feb738b48a16eb3dca278a65a29e1f"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.352512 26127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "540cf6e419e341af90d64164d5e2f16d"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "b5dfe22fcc0078b86a61fb8a9fd28415"
server_key_iv: "f5feb738b48a16eb3dca278a65a29e1f"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.357507 26127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:42:04.360844 26143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:04.362102 26127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:04.362236 26127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "540cf6e419e341af90d64164d5e2f16d"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "b5dfe22fcc0078b86a61fb8a9fd28415"
server_key_iv: "f5feb738b48a16eb3dca278a65a29e1f"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.362392 26127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:04.376760 26127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:04.381906 26127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:04.382302 26127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:04.388131 26127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:04.388263 26127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:04.388414 26127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:04.388530 26127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:42:04 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:04.417593 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.390149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:37375 (local address 127.23.130.62:36893)
0808 15:42:04.395383 (+ 5234us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:04.395387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:04.395404 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:04.395460 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:04.395464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:04.395548 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:04.395684 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:04.398608 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.399668 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.403101 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.403872 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.404025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.409915 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:04.409920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:04.409924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:04.409958 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:04.413672 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.414837 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.414843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.414848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.415057 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.415582 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.415588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.415591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.416014 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:04.416182 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:04.416972 (+ 790us) server_negotiation.cc:299] Negotiation successful
0808 15:42:04.417212 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5124,"thread_start_us":4986,"threads_started":1}
I20260808 15:42:04.419116 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.389859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.1:37375)
0808 15:42:04.390885 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:04.390934 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:04.394561 (+ 3627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:04.396051 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:04.396064 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:04.397266 (+ 1202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:04.398354 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.398388 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.399857 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.399866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.402846 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.402871 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.404427 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.404442 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.404822 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.406180 (+ 1358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:04.406204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:04.409587 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:04.413833 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.413845 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.413863 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.414671 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.415201 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.415208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.415217 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.415448 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.416160 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:04.416171 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:04.416583 (+ 412us) client_negotiation.cc:769] Sending connection context
0808 15:42:04.416865 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 15:42:04.417472 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":487,"threads_started":1}
I20260808 15:42:04.441636 26127 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:46463
I20260808 15:42:04.441696 26282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:46463 every 8 connection(s)
I20260808 15:42:04.443066 26127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:04.443470 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.425849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:42667 (local address 127.23.130.62:36893)
0808 15:42:04.426059 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:04.426067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:04.426842 (+ 775us) server_negotiation.cc:407] Connection header received
0808 15:42:04.427081 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:04.427086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:04.427166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:04.427245 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:04.428631 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.429759 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.432533 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.433304 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.433452 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.435989 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:04.435999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:04.436007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:04.436059 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:04.440043 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.441069 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.441077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.441086 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.441225 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.441684 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.441694 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.441696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.442059 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:04.442230 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:04.442773 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 15:42:04.443026 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:42:04.443598 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.425900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.1:42667)
0808 15:42:04.426726 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:04.426755 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:04.426945 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:04.427306 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:04.427314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:04.427926 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:04.428424 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.428441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.430029 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.430037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.432315 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.432339 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.433517 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.433523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.433648 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.434382 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:04.434445 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:04.435735 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:04.440220 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.440229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.440235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.440918 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.441354 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.441362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.441370 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.441524 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.442196 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:04.442202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:04.442465 (+ 263us) client_negotiation.cc:769] Sending connection context
0808 15:42:04.442646 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 15:42:04.443026 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":380,"threads_started":1}
I20260808 15:42:04.445693 26283 heartbeater.cc:347] Connected to a master server at 127.23.130.62:36893
I20260808 15:42:04.446808 26283 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:04.447434 26283 heartbeater.cc:511] Master 127.23.130.62:36893 requested a full tablet report, sending...
I20260808 15:42:04.449393 26049 ts_manager.cc:194] Registered new tserver with Master: 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463)
I20260808 15:42:04.450670 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26127
I20260808 15:42:04.450908 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:04.451124 26049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:37375
I20260808 15:42:04.451602 24072 external_mini_cluster.cc:1428] Setting key 9ff5c805e62a5292404bd1a0b5f8ae3f
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:04.498198 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:36893
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:04.837317 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:04.837783 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:04.837998 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:04.850028 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:04.850171 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:04.850323 26287 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:42:04.850420 26287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:04.850531 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:04.860617 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:04.861897 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:04.863654 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:04.877007 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:04.877058 26294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:04.877785 26296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:04.878221 26287 server_base.cc:1034] running on GCE node
I20260808 15:42:04.878931 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:04.879853 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:04.881079 26287 hybrid_clock.cc:648] HybridClock initialized: now 1786203724881016 us; error 55 us; skew 500 ppm
I20260808 15:42:04.881479 26287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:04 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:04.884766 26287 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:04.886207 26287 webserver.cc:460] Webserver started at http://127.23.130.2:41933/ using document root <none> and password file <none>
I20260808 15:42:04.886960 26287 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:04.887113 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:04.887449 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:04.890627 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "9da376c883a2405f97218e7efd2f5756"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "2a341352fcb8b5af2162a3f5625598d1"
server_key_iv: "8fd0197782c756dfb9143f14414f39b2"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.891402 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "9da376c883a2405f97218e7efd2f5756"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "2a341352fcb8b5af2162a3f5625598d1"
server_key_iv: "8fd0197782c756dfb9143f14414f39b2"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.896661 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:04.900254 26303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:04.901618 26287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:04.901804 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "9da376c883a2405f97218e7efd2f5756"
format_stamp: "Formatted at 2026-08-08 15:42:04 on dist-test-slave-gp6n"
server_key: "2a341352fcb8b5af2162a3f5625598d1"
server_key_iv: "8fd0197782c756dfb9143f14414f39b2"
server_key_version: "encryptionkey@0"
I20260808 15:42:04.902015 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:04.917073 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:04.921341 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:04.921630 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:04.924417 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:04.924566 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:04.924768 26287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:04.924903 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:42:04 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:04.951653 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.925430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.2:44235)
0808 15:42:04.926429 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:04.926481 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:04.929398 (+ 2917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:04.930648 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:04.930665 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:04.931638 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:04.932725 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.932765 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.934207 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.934213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.936226 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.936239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.937359 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.937368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.937586 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.938273 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:04.938296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:04.942091 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:04.946048 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.946062 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.946085 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.947002 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.947553 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.947565 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.947569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.947876 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.948464 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:04.948473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:04.948916 (+ 443us) client_negotiation.cc:769] Sending connection context
0808 15:42:04.949357 (+ 441us) client_negotiation.cc:240] Negotiation successful
0808 15:42:04.949989 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":347,"threads_started":1}
I20260808 15:42:04.953128 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.924754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:44235 (local address 127.23.130.62:36893)
0808 15:42:04.924979 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:04.924986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:04.926604 (+ 1618us) server_negotiation.cc:407] Connection header received
0808 15:42:04.929783 (+ 3179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:04.929791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:04.929925 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:04.930019 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:04.932976 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.933965 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.936389 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.937079 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.937238 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.942364 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:04.942370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:04.942377 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:04.942453 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:04.945901 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.947207 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.947214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.947218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.947381 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.948008 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.948019 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.948022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.948312 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:04.948497 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:04.952495 (+ 3998us) server_negotiation.cc:299] Negotiation successful
0808 15:42:04.952741 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:42:04.975822 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.961490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:50613 (local address 127.23.130.62:36893)
0808 15:42:04.961651 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:04.961655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:04.962285 (+ 630us) server_negotiation.cc:407] Connection header received
0808 15:42:04.962511 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:04.962518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:04.962581 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:04.962681 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:04.963644 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.964348 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.965773 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.966312 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.966493 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.969036 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:04.969046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:04.969052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:04.969090 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:04.972848 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.973751 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.973758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.973764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.973908 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:04.974277 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:04.974283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:04.974286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:04.974684 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:04.974842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:04.975244 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 15:42:04.975467 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:42:04.975955 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:04.961580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.2:50613)
0808 15:42:04.962181 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:04.962199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:04.962354 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:04.962715 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:04.962720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:04.963177 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:04.963492 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.963501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.964494 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.964499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.965632 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:04.965644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:04.966569 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:04.966584 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:04.966859 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:04.967655 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:04.967675 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:04.968821 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:04.972982 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.972987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.972991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.973619 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.974025 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:04.974030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:04.974034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:04.974167 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:04.974805 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:04.974810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:04.975002 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 15:42:04.975149 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:42:04.975473 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":346,"threads_started":1}
I20260808 15:42:04.980700 26287 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41545
I20260808 15:42:04.980746 26435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41545 every 8 connection(s)
I20260808 15:42:04.981930 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:04.983942 26436 heartbeater.cc:347] Connected to a master server at 127.23.130.62:36893
I20260808 15:42:04.984126 26436 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:04.984709 26436 heartbeater.cc:511] Master 127.23.130.62:36893 requested a full tablet report, sending...
I20260808 15:42:04.986012 26049 ts_manager.cc:194] Registered new tserver with Master: 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545)
I20260808 15:42:04.986884 26049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:44235
I20260808 15:42:04.988113 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26287
I20260808 15:42:04.988319 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:04.988763 24072 external_mini_cluster.cc:1428] Setting key 001e3978d6929f850b4889df487fb2fb
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:05.038300 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:36893
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:05.374711 26441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:05.374989 26441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:05.375159 26441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:05.382263 26441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:05.382371 26441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:05.382582 26441 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:42:05.382668 26441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:05.382797 26441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:05.392637 26441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:05.393970 26441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:05.395375 26441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:05.410820 26447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:05.410817 26446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:05.411124 26449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:05.411969 26441 server_base.cc:1034] running on GCE node
I20260808 15:42:05.412776 26441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:05.413831 26441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:05.415117 26441 hybrid_clock.cc:648] HybridClock initialized: now 1786203725415045 us; error 57 us; skew 500 ppm
I20260808 15:42:05.415473 26441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:05 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:05.418998 26441 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:05.420518 26441 webserver.cc:460] Webserver started at http://127.23.130.3:39701/ using document root <none> and password file <none>
I20260808 15:42:05.421237 26441 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:05.421399 26441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:05.421792 26441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:05.424842 26441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "1f54f48ba6d84607b613fde5128904b8"
format_stamp: "Formatted at 2026-08-08 15:42:05 on dist-test-slave-gp6n"
server_key: "9cab4da47b19a0e326e55cdf22fd9960"
server_key_iv: "6081252a62f41ba609fbf4f0555df3c4"
server_key_version: "encryptionkey@0"
I20260808 15:42:05.425709 26441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "1f54f48ba6d84607b613fde5128904b8"
format_stamp: "Formatted at 2026-08-08 15:42:05 on dist-test-slave-gp6n"
server_key: "9cab4da47b19a0e326e55cdf22fd9960"
server_key_iv: "6081252a62f41ba609fbf4f0555df3c4"
server_key_version: "encryptionkey@0"
I20260808 15:42:05.431092 26441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 15:42:05.434690 26458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:05.436071 26441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:42:05.436252 26441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "1f54f48ba6d84607b613fde5128904b8"
format_stamp: "Formatted at 2026-08-08 15:42:05 on dist-test-slave-gp6n"
server_key: "9cab4da47b19a0e326e55cdf22fd9960"
server_key_iv: "6081252a62f41ba609fbf4f0555df3c4"
server_key_version: "encryptionkey@0"
I20260808 15:42:05.436467 26441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:05.454167 26283 heartbeater.cc:503] Master 127.23.130.62:36893 was elected leader, sending a full tablet report...
I20260808 15:42:05.479523 26441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:05.484036 26441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:05.484365 26441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:05.487370 26441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:05.487500 26441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:05.487625 26441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:05.487746 26441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:42:05 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:05.513037 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.487987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:58149 (local address 127.23.130.62:36893)
0808 15:42:05.488805 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.488809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.489759 (+ 950us) server_negotiation.cc:407] Connection header received
0808 15:42:05.492494 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.492501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.492594 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.492739 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:05.495266 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.496384 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.498938 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.499395 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.499539 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.505038 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:05.505048 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:05.505055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:05.505110 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:05.509092 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.510114 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.510119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.510123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.510333 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.511041 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.511044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.511046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.511325 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:05.511437 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.512582 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.512750 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":581,"threads_started":1}
I20260808 15:42:05.515350 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.488725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.3:58149)
0808 15:42:05.489599 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.489638 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.492187 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.493210 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.493223 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.494047 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:05.495056 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.495090 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.496570 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.496575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.498775 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.498787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.499637 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.499648 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.500014 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.500869 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:05.500910 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:05.504671 (+ 3761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:05.509249 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.509258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.509285 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.509978 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.510519 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.510532 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.510538 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.510889 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.511463 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:05.511475 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:05.512089 (+ 614us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.512481 (+ 392us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.513209 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":387,"threads_started":1}
I20260808 15:42:05.536051 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.521320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:32837 (local address 127.23.130.62:36893)
0808 15:42:05.521508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.521515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.522156 (+ 641us) server_negotiation.cc:407] Connection header received
0808 15:42:05.522299 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.522308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.522424 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.522550 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:05.523917 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.525032 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.527231 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.527757 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.527904 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.529759 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:05.529765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:05.529770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:05.529810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:05.533059 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.533759 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.533764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.533767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.533981 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.534493 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.534501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.534506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.534871 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:05.535032 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.535462 (+ 430us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.535684 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:42:05.536159 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.521365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.23.130.3:32837)
0808 15:42:05.522056 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.522076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.522185 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.522598 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.522604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.523217 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:05.523770 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.523782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.525168 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.525196 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.527088 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.527100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.527956 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.527962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.528153 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.528812 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:05.528832 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:05.529578 (+ 746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:05.533185 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.533190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.533194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.533622 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.534135 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.534142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.534150 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.534304 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.534991 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:05.534997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:05.535196 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.535360 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.535713 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":360,"threads_started":1}
I20260808 15:42:05.547502 26441 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:36879
I20260808 15:42:05.547546 26600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:36879 every 8 connection(s)
I20260808 15:42:05.548751 26441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:05.550395 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26441
I20260808 15:42:05.550683 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:05.551017 26601 heartbeater.cc:347] Connected to a master server at 127.23.130.62:36893
I20260808 15:42:05.551246 26601 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:05.551294 24072 external_mini_cluster.cc:1428] Setting key b681678e51338ac90ccf76f508d7b34a
I20260808 15:42:05.552044 26601 heartbeater.cc:511] Master 127.23.130.62:36893 requested a full tablet report, sending...
I20260808 15:42:05.553435 26049 ts_manager.cc:194] Registered new tserver with Master: 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:05.554260 26049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:58149
I20260808 15:42:05.554610 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:42:05 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:42:05 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:05.607698 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.592912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.23.130.62:36893)
0808 15:42:05.593176 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.593181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.593285 (+ 104us) server_negotiation.cc:407] Connection header received
0808 15:42:05.593492 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.593497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.593566 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.593663 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:05.595225 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.595977 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.597235 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.597674 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.597782 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.601697 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:05.601702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:05.601706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:05.601741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:05.604764 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.605597 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.605602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.605604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.605739 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.606127 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.606131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.606133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.606585 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:05.606793 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.607091 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.607261 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:42:05.612130 26049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:05.619411 26049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:05.661588 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.632328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46463 (local address 127.0.0.1:33494)
0808 15:42:05.647475 (+ 15147us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.647535 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.647769 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.648719 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.648727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.648761 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:05.649180 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.649191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.652029 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.652040 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.656521 (+ 4481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.656546 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.659912 (+ 3366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.659923 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.660297 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.660395 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.660589 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.660888 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14800,"thread_start_us":15692,"threads_started":1}
I20260808 15:42:05.662909 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.632662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.23.130.1:46463)
0808 15:42:05.645641 (+ 12979us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.645655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.647653 (+ 1998us) server_negotiation.cc:407] Connection header received
0808 15:42:05.648081 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.648091 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.648401 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.648618 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:05.649391 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.651746 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.656780 (+ 5034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.659138 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.659302 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.659636 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.662126 (+ 2490us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.662447 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12734,"thread_start_us":12528,"threads_started":1}
I20260808 15:42:05.668226 26204 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7d32a467e44739375c75bada1b93c (DEFAULT_TABLE table=test-table [id=04827e2cb50244aeb04a8258bd6be2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:05.669960 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7d32a467e44739375c75bada1b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.670576 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.633112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41545 (local address 127.0.0.1:58408)
0808 15:42:05.662235 (+ 29123us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.662256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.662442 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.664060 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.664068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.664103 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:05.664526 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.664541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.666547 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.666556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.668083 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.668095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.669504 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.669514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.669754 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.669799 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.669928 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.670093 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28973,"mutex_wait_us":1,"thread_start_us":31816,"threads_started":1}
I20260808 15:42:05.671516 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.633613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.23.130.2:41545)
0808 15:42:05.661533 (+ 27920us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.661542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.662613 (+ 1071us) server_negotiation.cc:407] Connection header received
0808 15:42:05.662761 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.662774 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.663118 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.663371 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:05.664805 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.666334 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.668244 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.669253 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.669414 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.669625 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.671119 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.671306 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27721,"thread_start_us":27941,"threads_started":1}
I20260808 15:42:05.675776 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.633278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.0.0.1:56750)
0808 15:42:05.663695 (+ 30417us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.663716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.663823 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.666501 (+ 2678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.666507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.666537 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:05.666800 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.666824 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.669268 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.669291 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.671081 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.671097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.674994 (+ 3897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.675000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.675188 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.675221 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.675359 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.675501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30317,"thread_start_us":31409,"threads_started":1}
I20260808 15:42:05.675350 26357 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7d32a467e44739375c75bada1b93c (DEFAULT_TABLE table=test-table [id=04827e2cb50244aeb04a8258bd6be2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:05.675957 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.633633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.23.130.3:36879)
0808 15:42:05.665701 (+ 32068us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.665710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.665778 (+ 68us) server_negotiation.cc:407] Connection header received
0808 15:42:05.665899 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.665909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.666216 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.667002 (+ 786us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:05.667360 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.669100 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.671268 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.674812 (+ 3544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.674979 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.675211 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.675414 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.675574 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31870,"thread_start_us":31669,"threads_started":1}
I20260808 15:42:05.676987 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7d32a467e44739375c75bada1b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.678953 26517 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7d32a467e44739375c75bada1b93c (DEFAULT_TABLE table=test-table [id=04827e2cb50244aeb04a8258bd6be2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:05.680051 26517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7d32a467e44739375c75bada1b93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.686501 26622 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Bootstrap starting.
I20260808 15:42:05.690280 26622 tablet_bootstrap.cc:654] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.691613 26622 log.cc:824] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:05.691987 26623 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Bootstrap starting.
I20260808 15:42:05.693943 26625 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Bootstrap starting.
I20260808 15:42:05.695770 26622 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: No bootstrap required, opened a new log
I20260808 15:42:05.696075 26622 ts_tablet_manager.cc:1397] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 15:42:05.696590 26623 tablet_bootstrap.cc:654] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.696947 26625 tablet_bootstrap.cc:654] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.697892 26625 log.cc:824] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:05.697922 26623 log.cc:824] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:05.700709 26623 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: No bootstrap required, opened a new log
I20260808 15:42:05.700999 26623 ts_tablet_manager.cc:1397] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 15:42:05.701109 26625 tablet_bootstrap.cc:492] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: No bootstrap required, opened a new log
I20260808 15:42:05.701383 26625 ts_tablet_manager.cc:1397] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260808 15:42:05.708031 26622 raft_consensus.cc:348] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.708541 26622 raft_consensus.cc:374] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.708712 26622 raft_consensus.cc:729] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.708531 26623 raft_consensus.cc:348] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.708956 26623 raft_consensus.cc:374] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.709117 26623 raft_consensus.cc:729] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.709425 26622 consensus_queue.cc:260] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.709776 26623 consensus_queue.cc:260] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.712921 26623 ts_tablet_manager.cc:1428] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260808 15:42:05.713317 26622 ts_tablet_manager.cc:1428] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260808 15:42:05.713735 26625 raft_consensus.cc:348] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.714154 26436 heartbeater.cc:503] Master 127.23.130.62:36893 was elected leader, sending a full tablet report...
I20260808 15:42:05.714217 26625 raft_consensus.cc:374] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.714334 26625 raft_consensus.cc:729] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.715016 26625 consensus_queue.cc:260] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.716389 26601 heartbeater.cc:503] Master 127.23.130.62:36893 was elected leader, sending a full tablet report...
I20260808 15:42:05.717010 26625 ts_tablet_manager.cc:1428] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
W20260808 15:42:05.734035 26437 tablet.cc:2369] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:42:05.801189 26602 tablet.cc:2369] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:05.861177 26629 raft_consensus.cc:482] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:05.861594 26629 raft_consensus.cc:504] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.863308 26629 leader_election.cc:288] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:05.870375 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.863846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:37619 (local address 127.23.130.2:41545)
0808 15:42:05.864052 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.864058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.864574 (+ 516us) server_negotiation.cc:407] Connection header received
0808 15:42:05.864828 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.864832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.864893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.864999 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:05.865514 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.866285 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.868360 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.869449 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.869575 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.869629 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.870001 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.870099 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:42:05.870556 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.863761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41545 (local address 127.23.130.1:37619)
0808 15:42:05.864456 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.864471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.864656 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.865014 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.865018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.865042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:05.865363 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.865372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.866442 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.866449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.868177 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.868190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.869655 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.869661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.869826 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.869864 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.869970 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.870102 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":23,"thread_start_us":397,"threads_started":1}
I20260808 15:42:05.871277 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.863759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:48569 (local address 127.23.130.3:36879)
0808 15:42:05.863974 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.863981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.865030 (+ 1049us) server_negotiation.cc:407] Connection header received
0808 15:42:05.865280 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.865289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.865376 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.865479 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:05.866136 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.867019 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.869135 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.870203 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.870365 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.870471 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.870901 (+ 430us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.871030 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:42:05.871348 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.863803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.23.130.1:48569)
0808 15:42:05.864911 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.864925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.865082 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.865492 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.865496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.865544 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:05.865948 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.865960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.867161 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.867167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.868985 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.869000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.870503 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.870514 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.870697 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.870737 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.870873 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.870994 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":1,"thread_start_us":832,"threads_started":1}
I20260808 15:42:05.871717 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7d32a467e44739375c75bada1b93c" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756" is_pre_election: true
I20260808 15:42:05.872347 26378 raft_consensus.cc:2454] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 0.
I20260808 15:42:05.872303 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7d32a467e44739375c75bada1b93c" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8" is_pre_election: true
I20260808 15:42:05.872903 26545 raft_consensus.cc:2454] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 0.
I20260808 15:42:05.873136 26146 leader_election.cc:302] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:05.873560 26629 raft_consensus.cc:2781] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:05.873723 26629 raft_consensus.cc:482] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:05.873832 26629 raft_consensus.cc:3037] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:05.876032 26629 raft_consensus.cc:504] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.876642 26629 leader_election.cc:288] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 election: Requested vote from peers 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:05.877058 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7d32a467e44739375c75bada1b93c" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756"
I20260808 15:42:05.877213 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7d32a467e44739375c75bada1b93c" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8"
I20260808 15:42:05.877278 26378 raft_consensus.cc:3037] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:05.877434 26545 raft_consensus.cc:3037] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:05.880582 26545 raft_consensus.cc:2454] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 1.
I20260808 15:42:05.880582 26378 raft_consensus.cc:2454] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 1.
I20260808 15:42:05.880991 26146 leader_election.cc:302] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:05.881285 26629 raft_consensus.cc:2781] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:05.881740 26629 raft_consensus.cc:686] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 1 LEADER]: Becoming Leader. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Running, Role: LEADER
I20260808 15:42:05.882220 26629 consensus_queue.cc:237] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.886685 26049 catalog_manager.cc:5495] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d reported cstate change: term changed from 0 to 1, leader changed from <none> to 540cf6e419e341af90d64164d5e2f16d (127.23.130.1). New cstate: current_term: 1 leader_uuid: "540cf6e419e341af90d64164d5e2f16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } health_report { overall_health: UNKNOWN } } }
W20260808 15:42:05.946740 26284 tablet.cc:2369] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:42:05 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203723, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:05.962595 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.949004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:36893 (local address 127.0.0.1:60148)
0808 15:42:05.949592 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:05.949608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:05.949736 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:05.950165 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:05.950168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:05.950959 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:05.951235 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.951244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.952473 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.952477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.953509 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:05.953521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.954271 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.954276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:05.954468 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.955181 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:05.955212 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:05.957416 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:05.960234 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.960244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.960250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.960678 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.961011 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:05.961014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:05.961016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:05.961168 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:05.961642 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:05.961648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:05.961798 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 15:42:05.961949 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 15:42:05.962159 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":349,"threads_started":1}
I20260808 15:42:05.962634 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:05.948911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.23.130.62:36893)
0808 15:42:05.949120 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:05.949129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:05.949705 (+ 576us) server_negotiation.cc:407] Connection header received
0808 15:42:05.949901 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:05.949906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:05.949999 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:05.950118 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:05.951533 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.952281 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.953650 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:05.954083 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:05.954176 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:05.957587 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:05.957592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:05.957595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:05.957628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:05.960113 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.960784 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.960788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.960790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.960917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:05.961261 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:05.961264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:05.961266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:05.961498 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:05.961604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:05.962056 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 15:42:05.962279 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":69}
I20260808 15:42:05.967871 26049 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:60148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:42:05.968824 26049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:05.975605 26204 tablet_service.cc:1467] Processing CreateTablet for tablet a216bc0fd87d42959564619bb113e9d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76bba216f494790b0d45b5c8da956d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:05.975689 26357 tablet_service.cc:1467] Processing CreateTablet for tablet a216bc0fd87d42959564619bb113e9d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76bba216f494790b0d45b5c8da956d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:05.975935 26517 tablet_service.cc:1467] Processing CreateTablet for tablet a216bc0fd87d42959564619bb113e9d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76bba216f494790b0d45b5c8da956d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:05.976593 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a216bc0fd87d42959564619bb113e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.976627 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a216bc0fd87d42959564619bb113e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.976653 26517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a216bc0fd87d42959564619bb113e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:05.983110 26622 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: Bootstrap starting.
I20260808 15:42:05.983759 26623 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: Bootstrap starting.
I20260808 15:42:05.983752 26625 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: Bootstrap starting.
I20260808 15:42:05.984763 26622 tablet_bootstrap.cc:654] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.985419 26625 tablet_bootstrap.cc:654] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.985440 26623 tablet_bootstrap.cc:654] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:05.996008 26622 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: No bootstrap required, opened a new log
I20260808 15:42:05.996227 26622 ts_tablet_manager.cc:1397] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:42:05.996488 26623 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: No bootstrap required, opened a new log
I20260808 15:42:05.996538 26625 tablet_bootstrap.cc:492] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: No bootstrap required, opened a new log
I20260808 15:42:05.996659 26623 ts_tablet_manager.cc:1397] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260808 15:42:05.996707 26625 ts_tablet_manager.cc:1397] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260808 15:42:05.997165 26622 raft_consensus.cc:348] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.997491 26622 raft_consensus.cc:374] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.997635 26622 raft_consensus.cc:729] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.997468 26625 raft_consensus.cc:348] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.997488 26623 raft_consensus.cc:348] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.997730 26625 raft_consensus.cc:374] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.997741 26623 raft_consensus.cc:374] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:05.997843 26623 raft_consensus.cc:729] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.997889 26625 raft_consensus.cc:729] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Initialized, Role: FOLLOWER
I20260808 15:42:05.998013 26622 consensus_queue.cc:260] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.998160 26625 consensus_queue.cc:260] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.998189 26623 consensus_queue.cc:260] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:05.998829 26623 ts_tablet_manager.cc:1428] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:05.998837 26625 ts_tablet_manager.cc:1428] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:05.998713 26630 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:05.998699 26631 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:05.999099 26631 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:05.999118 26630 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.000383 26629 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.000670 26629 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.001993 26622 ts_tablet_manager.cc:1428] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260808 15:42:06.044785 26631 raft_consensus.cc:482] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:06.044991 26631 raft_consensus.cc:504] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.046518 26631 leader_election.cc:288] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545)
I20260808 15:42:06.052371 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.046856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:53879 (local address 127.23.130.1:46463)
0808 15:42:06.047054 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.047061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.047402 (+ 341us) server_negotiation.cc:407] Connection header received
0808 15:42:06.047615 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.047620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.047691 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.047788 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:06.048225 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.048943 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.050659 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.051437 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.051612 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.051689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.051988 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.052116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:42:06.052537 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.046750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46463 (local address 127.23.130.3:53879)
0808 15:42:06.047286 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:06.047306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:06.047454 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:06.047806 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:06.047810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:06.047843 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:06.048064 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.048074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.049069 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.049077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.050510 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.050522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.051611 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.051617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.051804 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.051830 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:42:06.051960 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 15:42:06.052095 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1,"thread_start_us":315,"threads_started":1}
I20260808 15:42:06.052690 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.047013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:32829 (local address 127.23.130.2:41545)
0808 15:42:06.047188 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.047196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.047530 (+ 334us) server_negotiation.cc:407] Connection header received
0808 15:42:06.047723 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.047727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.047786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.047883 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:06.048416 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.049147 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.050870 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.051629 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.051810 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.051886 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.052240 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.052381 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 15:42:06.053190 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.046781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41545 (local address 127.23.130.3:32829)
0808 15:42:06.047416 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:06.047435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:06.047574 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:06.047906 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:06.047910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:06.047956 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:06.048246 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.048254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.049295 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.049304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.050727 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.050739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.051857 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.051863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.052025 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.052082 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:42:06.052740 (+ 658us) client_negotiation.cc:240] Negotiation successful
0808 15:42:06.052889 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":410,"threads_started":1}
I20260808 15:42:06.053556 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a216bc0fd87d42959564619bb113e9d0" candidate_uuid: "1f54f48ba6d84607b613fde5128904b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d" is_pre_election: true
I20260808 15:42:06.053992 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a216bc0fd87d42959564619bb113e9d0" candidate_uuid: "1f54f48ba6d84607b613fde5128904b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756" is_pre_election: true
I20260808 15:42:06.054034 26227 raft_consensus.cc:2454] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f54f48ba6d84607b613fde5128904b8 in term 0.
I20260808 15:42:06.054265 26378 raft_consensus.cc:2454] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f54f48ba6d84607b613fde5128904b8 in term 0.
I20260808 15:42:06.054761 26461 leader_election.cc:302] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f54f48ba6d84607b613fde5128904b8, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:06.055197 26631 raft_consensus.cc:2781] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:06.055351 26631 raft_consensus.cc:482] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:06.055439 26631 raft_consensus.cc:3037] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.057703 26631 raft_consensus.cc:504] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.058317 26631 leader_election.cc:288] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [CANDIDATE]: Term 1 election: Requested vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545)
I20260808 15:42:06.058797 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a216bc0fd87d42959564619bb113e9d0" candidate_uuid: "1f54f48ba6d84607b613fde5128904b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756"
I20260808 15:42:06.058820 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a216bc0fd87d42959564619bb113e9d0" candidate_uuid: "1f54f48ba6d84607b613fde5128904b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d"
I20260808 15:42:06.059010 26378 raft_consensus.cc:3037] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.059049 26227 raft_consensus.cc:3037] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.061017 26378 raft_consensus.cc:2454] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f54f48ba6d84607b613fde5128904b8 in term 1.
I20260808 15:42:06.061173 26227 raft_consensus.cc:2454] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f54f48ba6d84607b613fde5128904b8 in term 1.
I20260808 15:42:06.061457 26461 leader_election.cc:302] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f54f48ba6d84607b613fde5128904b8, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:06.061769 26631 raft_consensus.cc:2781] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:06.062148 26631 raft_consensus.cc:686] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Running, Role: LEADER
I20260808 15:42:06.062915 26631 consensus_queue.cc:237] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.066846 26651 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: TxnStatusTablet state changed. Reason: New leader 1f54f48ba6d84607b613fde5128904b8. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.067289 26651 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:06.068063 26653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:42:06.068357 26653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:42:06.068598 26047 catalog_manager.cc:5495] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } health_report { overall_health: HEALTHY } } }
I20260808 15:42:06.087772 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.082007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.0.0.1:56764)
0808 15:42:06.082288 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:06.082305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:06.082493 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:06.082963 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:06.082967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:06.082980 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:06.083176 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.083186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.084303 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.084310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.085322 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.085334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.086063 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.086069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.086278 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.086340 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.087213 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.087235 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:42:06.087321 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 15:42:06.087470 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 15:42:06.087819 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.082012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.23.130.3:36879)
0808 15:42:06.082193 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.082199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.082450 (+ 251us) server_negotiation.cc:407] Connection header received
0808 15:42:06.082709 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.082713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.082795 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.082943 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:06.083321 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.084157 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.085468 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.085894 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.086029 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.086582 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.087089 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.087259 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.087352 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.087436 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:42:06.096457 26227 raft_consensus.cc:1261] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Refusing update from remote peer 1f54f48ba6d84607b613fde5128904b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.096571 26378 raft_consensus.cc:1261] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Refusing update from remote peer 1f54f48ba6d84607b613fde5128904b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.097483 26651 consensus_queue.cc:1035] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.097815 26631 consensus_queue.cc:1035] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.109467 26629 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: TxnStatusTablet state changed. Reason: New leader 1f54f48ba6d84607b613fde5128904b8. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.109892 26629 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.111039 26630 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: TxnStatusTablet state changed. Reason: New leader 1f54f48ba6d84607b613fde5128904b8. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.111331 26630 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.115403 26656 mvcc.cc:205] Tried to move back new op lower bound from 7316290462080212992 to 7316290461958426624. Current Snapshot: MvccSnapshot[applied={T|T < 7316290462080212992}]
I20260808 15:42:06.123092 26651 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.123392 26651 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:06.127345 26651 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.127612 26651 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 1f54f48ba6d84607b613fde5128904b8: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:06.132407 26630 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.132668 26630 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 9da376c883a2405f97218e7efd2f5756: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.134665 26629 tablet_replica.cc:439] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f54f48ba6d84607b613fde5128904b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } } }
I20260808 15:42:06.134948 26629 tablet_replica.cc:442] T a216bc0fd87d42959564619bb113e9d0 P 540cf6e419e341af90d64164d5e2f16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:06.144531 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.137613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.23.130.1:46463)
0808 15:42:06.137798 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.137804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.138222 (+ 418us) server_negotiation.cc:407] Connection header received
0808 15:42:06.138377 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.138382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.138529 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.138691 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:06.139103 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.140314 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.141821 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.142503 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.142628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.143185 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.143877 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.144025 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.144104 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.144195 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:42:06.162042 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.155489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.23.130.1:59869)
0808 15:42:06.156205 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:06.156232 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:06.156389 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:06.156813 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:06.156816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:06.156827 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:06.157145 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.157154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.158678 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.158683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.159783 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.159796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.160842 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.160847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.160975 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.161041 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.161542 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.161552 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:42:06.161616 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 15:42:06.161726 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"spinlock_wait_cycles":8448,"thread_start_us":459,"threads_started":1}
I20260808 15:42:06.162046 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.155908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:59869 (local address 127.23.130.3:36879)
0808 15:42:06.156119 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.156126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.156310 (+ 184us) server_negotiation.cc:407] Connection header received
0808 15:42:06.156546 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.156552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.156663 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.156770 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:06.157319 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.158549 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.159927 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.160626 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.160770 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.161282 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.161441 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.161534 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.161665 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.161750 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:42:06.173120 26378 raft_consensus.cc:1261] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Refusing update from remote peer 540cf6e419e341af90d64164d5e2f16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.173278 26545 raft_consensus.cc:1261] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Refusing update from remote peer 540cf6e419e341af90d64164d5e2f16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.173884 26635 consensus_queue.cc:1035] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.174578 26629 consensus_queue.cc:1035] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.186234 26657 mvcc.cc:205] Tried to move back new op lower bound from 7316290462398230528 to 7316290461217329152. Current Snapshot: MvccSnapshot[applied={T|T < 7316290462398230528}]
I20260808 15:42:06.223816 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.210520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:45811 (local address 127.23.130.1:46463)
0808 15:42:06.210714 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:06.210721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:06.215172 (+ 4451us) server_negotiation.cc:407] Connection header received
0808 15:42:06.215390 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:06.215394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:06.215472 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:06.215546 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:06.216187 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.217360 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.219617 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.220400 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.220554 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.221312 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.222743 (+ 1431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.222841 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:06.223227 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 15:42:06.223345 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:42:06.224798 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:06.210387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46463 (local address 127.23.130.3:45811)
0808 15:42:06.215081 (+ 4694us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:06.215102 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:06.215240 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:06.215587 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:06.215593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:06.215611 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:06.215996 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.216008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.217577 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.217582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.219403 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:06.219464 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:06.220641 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:06.220674 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:06.220950 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:06.221058 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:06.222976 (+ 1918us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:06.222994 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 15:42:06.224124 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0808 15:42:06.224356 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4502,"thread_start_us":4180,"threads_started":1}
I20260808 15:42:06.261264 26048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:06.261576 26048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:06.265630 26048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Sending DeleteTablet for 3 replicas of tablet a3f7d32a467e44739375c75bada1b93c
I20260808 15:42:06.266772 26204 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7d32a467e44739375c75bada1b93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:06 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:33494
I20260808 15:42:06.266860 26357 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7d32a467e44739375c75bada1b93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:06 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260808 15:42:06.268128 26517 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7d32a467e44739375c75bada1b93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:06 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:56750
I20260808 15:42:06.276619 26679 tablet_replica.cc:330] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: stopping tablet replica
I20260808 15:42:06.276816 26048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 15:42:06.277349 26679 raft_consensus.cc:2229] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:06.278136 26679 raft_consensus.cc:2258] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 15:42:06.278853 26048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:06.280189 26679 ts_tablet_manager.cc:1905] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:06.290529 26204 tablet_service.cc:1467] Processing CreateTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 (DEFAULT_TABLE table=test-table [id=28e41aefcff7412c9ae4c03d823f97b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:06.291016 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296f1912fd46426f839a3ee88ac6d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:06.292501 26679 ts_tablet_manager.cc:1918] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:06.292771 26679 log.cc:1196] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/a3f7d32a467e44739375c75bada1b93c
I20260808 15:42:06.293550 26679 ts_tablet_manager.cc:1939] T a3f7d32a467e44739375c75bada1b93c P 540cf6e419e341af90d64164d5e2f16d: Deleting consensus metadata
I20260808 15:42:06.295636 26033 catalog_manager.cc:4841] TS 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463): tablet a3f7d32a467e44739375c75bada1b93c (table test-table [id=04827e2cb50244aeb04a8258bd6be2c7]) successfully deleted
I20260808 15:42:06.295706 26516 tablet_service.cc:1467] Processing CreateTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 (DEFAULT_TABLE table=test-table [id=28e41aefcff7412c9ae4c03d823f97b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:06.296331 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296f1912fd46426f839a3ee88ac6d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:06.299149 26356 tablet_service.cc:1467] Processing CreateTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 (DEFAULT_TABLE table=test-table [id=28e41aefcff7412c9ae4c03d823f97b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:06.299685 26356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296f1912fd46426f839a3ee88ac6d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:06.302554 26622 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Bootstrap starting.
I20260808 15:42:06.304217 26622 tablet_bootstrap.cc:654] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:06.305732 26625 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Bootstrap starting.
I20260808 15:42:06.307503 26625 tablet_bootstrap.cc:654] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:06.311954 26623 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Bootstrap starting.
I20260808 15:42:06.312296 26681 tablet_replica.cc:330] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: stopping tablet replica
I20260808 15:42:06.312762 26681 raft_consensus.cc:2229] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:06.313112 26681 raft_consensus.cc:2258] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:06.313939 26623 tablet_bootstrap.cc:654] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:06.315212 26681 ts_tablet_manager.cc:1905] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:06.319315 26680 tablet_replica.cc:330] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: stopping tablet replica
I20260808 15:42:06.319736 26680 raft_consensus.cc:2229] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:06.320214 26680 raft_consensus.cc:2258] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:06.321619 26622 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: No bootstrap required, opened a new log
I20260808 15:42:06.321669 26625 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: No bootstrap required, opened a new log
I20260808 15:42:06.321861 26625 ts_tablet_manager.cc:1397] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20260808 15:42:06.321911 26622 ts_tablet_manager.cc:1397] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260808 15:42:06.322208 26680 ts_tablet_manager.cc:1905] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:06.324018 26625 raft_consensus.cc:348] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.324386 26625 raft_consensus.cc:374] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:06.324530 26625 raft_consensus.cc:729] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Initialized, Role: FOLLOWER
I20260808 15:42:06.324999 26625 consensus_queue.cc:260] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.325093 26622 raft_consensus.cc:348] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.325348 26622 raft_consensus.cc:374] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:06.325424 26622 raft_consensus.cc:729] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Initialized, Role: FOLLOWER
I20260808 15:42:06.326007 26625 ts_tablet_manager.cc:1428] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 15:42:06.326946 26622 consensus_queue.cc:260] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.329521 26623 tablet_bootstrap.cc:492] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: No bootstrap required, opened a new log
I20260808 15:42:06.329778 26623 ts_tablet_manager.cc:1397] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20260808 15:42:06.330884 26623 raft_consensus.cc:348] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.331203 26623 raft_consensus.cc:374] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:06.331355 26623 raft_consensus.cc:729] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Initialized, Role: FOLLOWER
I20260808 15:42:06.331749 26623 consensus_queue.cc:260] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.332059 26681 ts_tablet_manager.cc:1918] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:06.332285 26681 log.cc:1196] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/a3f7d32a467e44739375c75bada1b93c
I20260808 15:42:06.332844 26681 ts_tablet_manager.cc:1939] T a3f7d32a467e44739375c75bada1b93c P 1f54f48ba6d84607b613fde5128904b8: Deleting consensus metadata
I20260808 15:42:06.333252 26622 ts_tablet_manager.cc:1428] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260808 15:42:06.334631 26035 catalog_manager.cc:4841] TS 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879): tablet a3f7d32a467e44739375c75bada1b93c (table test-table [id=04827e2cb50244aeb04a8258bd6be2c7]) successfully deleted
I20260808 15:42:06.334722 26623 ts_tablet_manager.cc:1428] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 15:42:06.337723 26680 ts_tablet_manager.cc:1918] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:06.337908 26680 log.cc:1196] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/a3f7d32a467e44739375c75bada1b93c
I20260808 15:42:06.338451 26680 ts_tablet_manager.cc:1939] T a3f7d32a467e44739375c75bada1b93c P 9da376c883a2405f97218e7efd2f5756: Deleting consensus metadata
I20260808 15:42:06.340219 26034 catalog_manager.cc:4841] TS 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545): tablet a3f7d32a467e44739375c75bada1b93c (table test-table [id=04827e2cb50244aeb04a8258bd6be2c7]) successfully deleted
I20260808 15:42:06.439834 26674 raft_consensus.cc:482] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:06.440128 26674 raft_consensus.cc:504] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.441045 26674 leader_election.cc:288] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:06.442041 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296f1912fd46426f839a3ee88ac6d7a3" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756" is_pre_election: true
I20260808 15:42:06.442307 26378 raft_consensus.cc:2454] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 0.
I20260808 15:42:06.442903 26146 leader_election.cc:302] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:06.442950 26541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296f1912fd46426f839a3ee88ac6d7a3" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8" is_pre_election: true
I20260808 15:42:06.443226 26674 raft_consensus.cc:2781] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:06.443284 26541 raft_consensus.cc:2454] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 0.
I20260808 15:42:06.443344 26674 raft_consensus.cc:482] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:06.443461 26674 raft_consensus.cc:3037] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.446820 26674 raft_consensus.cc:504] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.447937 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296f1912fd46426f839a3ee88ac6d7a3" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da376c883a2405f97218e7efd2f5756"
I20260808 15:42:06.448146 26378 raft_consensus.cc:3037] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.451231 26378 raft_consensus.cc:2454] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 1.
I20260808 15:42:06.451694 26146 leader_election.cc:302] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:06.451995 26635 raft_consensus.cc:2781] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:06.452203 26635 raft_consensus.cc:686] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 1 LEADER]: Becoming Leader. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Running, Role: LEADER
I20260808 15:42:06.452543 26635 consensus_queue.cc:237] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:06.456103 26674 leader_election.cc:288] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [CANDIDATE]: Term 1 election: Requested vote from peers 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:06.456274 26541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296f1912fd46426f839a3ee88ac6d7a3" candidate_uuid: "540cf6e419e341af90d64164d5e2f16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8"
I20260808 15:42:06.456525 26541 raft_consensus.cc:3037] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:06.460366 26541 raft_consensus.cc:2454] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540cf6e419e341af90d64164d5e2f16d in term 1.
I20260808 15:42:06.465097 26047 catalog_manager.cc:5495] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d reported cstate change: term changed from 0 to 1, leader changed from <none> to 540cf6e419e341af90d64164d5e2f16d (127.23.130.1). New cstate: current_term: 1 leader_uuid: "540cf6e419e341af90d64164d5e2f16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:06.533825 26378 raft_consensus.cc:1261] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Refusing update from remote peer 540cf6e419e341af90d64164d5e2f16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.534001 26541 raft_consensus.cc:1261] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Refusing update from remote peer 540cf6e419e341af90d64164d5e2f16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:06.534500 26674 consensus_queue.cc:1035] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.535461 26674 consensus_queue.cc:1035] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:06.553977 26658 mvcc.cc:205] Tried to move back new op lower bound from 7316290463877058560 to 7316290463552540672. Current Snapshot: MvccSnapshot[applied={T|T < 7316290463877058560}]
W20260808 15:42:06.572276 26657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:42:08.165140 26048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:08.165457 26048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:08.169466 26048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Sending DeleteTablet for 3 replicas of tablet 296f1912fd46426f839a3ee88ac6d7a3
I20260808 15:42:08.170265 26204 tablet_service.cc:1514] Processing DeleteTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:33494
I20260808 15:42:08.170318 26356 tablet_service.cc:1514] Processing DeleteTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260808 15:42:08.170380 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet 296f1912fd46426f839a3ee88ac6d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:56750
I20260808 15:42:08.182989 26712 tablet_replica.cc:330] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: stopping tablet replica
I20260808 15:42:08.183470 26712 raft_consensus.cc:2229] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:08.184060 26712 raft_consensus.cc:2258] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:08.185472 26712 ts_tablet_manager.cc:1905] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.185626 26715 tablet_replica.cc:330] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: stopping tablet replica
I20260808 15:42:08.185832 26715 raft_consensus.cc:2229] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:08.186162 26715 raft_consensus.cc:2258] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:08.186849 26714 tablet_replica.cc:330] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: stopping tablet replica
I20260808 15:42:08.187067 26714 raft_consensus.cc:2229] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:08.187327 26715 ts_tablet_manager.cc:1905] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.187387 26714 raft_consensus.cc:2258] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:08.188386 26714 ts_tablet_manager.cc:1905] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.192432 26712 ts_tablet_manager.cc:1918] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:08.192620 26712 log.cc:1196] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/296f1912fd46426f839a3ee88ac6d7a3
I20260808 15:42:08.193113 26712 ts_tablet_manager.cc:1939] T 296f1912fd46426f839a3ee88ac6d7a3 P 540cf6e419e341af90d64164d5e2f16d: Deleting consensus metadata
I20260808 15:42:08.193979 26715 ts_tablet_manager.cc:1918] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:08.194183 26715 log.cc:1196] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/296f1912fd46426f839a3ee88ac6d7a3
I20260808 15:42:08.194396 26033 catalog_manager.cc:4841] TS 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463): tablet 296f1912fd46426f839a3ee88ac6d7a3 (table test-table [id=28e41aefcff7412c9ae4c03d823f97b2]) successfully deleted
I20260808 15:42:08.194722 26715 ts_tablet_manager.cc:1939] T 296f1912fd46426f839a3ee88ac6d7a3 P 1f54f48ba6d84607b613fde5128904b8: Deleting consensus metadata
I20260808 15:42:08.195360 26714 ts_tablet_manager.cc:1918] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:08.195508 26714 log.cc:1196] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/296f1912fd46426f839a3ee88ac6d7a3
I20260808 15:42:08.195829 26035 catalog_manager.cc:4841] TS 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879): tablet 296f1912fd46426f839a3ee88ac6d7a3 (table test-table [id=28e41aefcff7412c9ae4c03d823f97b2]) successfully deleted
I20260808 15:42:08.196007 26714 ts_tablet_manager.cc:1939] T 296f1912fd46426f839a3ee88ac6d7a3 P 9da376c883a2405f97218e7efd2f5756: Deleting consensus metadata
I20260808 15:42:08.196898 26034 catalog_manager.cc:4841] TS 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545): tablet 296f1912fd46426f839a3ee88ac6d7a3 (table test-table [id=28e41aefcff7412c9ae4c03d823f97b2]) successfully deleted
Aug 08 15:42:08 dist-test-slave-gp6n krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:42:08 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:08.240844 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.226283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.23.130.62:36893)
0808 15:42:08.226893 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.226902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.226981 (+ 79us) server_negotiation.cc:407] Connection header received
0808 15:42:08.227222 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.227232 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.227299 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.227415 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:08.228898 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.229713 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.231106 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.231581 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.231729 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.235532 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:08.235538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:08.235541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:08.235574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:08.238092 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:08.238837 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:08.238847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:08.238850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:08.239000 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:08.239406 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:08.239410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:08.239412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:08.239678 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:08.240004 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.240237 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.240463 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":293,"threads_started":1}
I20260808 15:42:08.244529 26048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:08.245208 26048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:08.251828 26204 tablet_service.cc:1467] Processing CreateTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 (DEFAULT_TABLE table=test-table [id=933564a584f6412b9af67bbe38f6121d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.252180 26516 tablet_service.cc:1467] Processing CreateTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 (DEFAULT_TABLE table=test-table [id=933564a584f6412b9af67bbe38f6121d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.252334 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3906fe0805364dca8a2b41e4e35420d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.252652 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3906fe0805364dca8a2b41e4e35420d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.252591 26356 tablet_service.cc:1467] Processing CreateTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 (DEFAULT_TABLE table=test-table [id=933564a584f6412b9af67bbe38f6121d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.253024 26356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3906fe0805364dca8a2b41e4e35420d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.258853 26724 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Bootstrap starting.
I20260808 15:42:08.260072 26726 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Bootstrap starting.
I20260808 15:42:08.260825 26725 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Bootstrap starting.
I20260808 15:42:08.261312 26724 tablet_bootstrap.cc:654] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.262070 26726 tablet_bootstrap.cc:654] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.262899 26725 tablet_bootstrap.cc:654] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.264057 26724 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: No bootstrap required, opened a new log
I20260808 15:42:08.264266 26724 ts_tablet_manager.cc:1397] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 15:42:08.264798 26725 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: No bootstrap required, opened a new log
I20260808 15:42:08.265007 26725 ts_tablet_manager.cc:1397] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:42:08.265316 26724 raft_consensus.cc:348] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.265622 26724 raft_consensus.cc:374] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.265735 26724 raft_consensus.cc:729] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.265934 26725 raft_consensus.cc:348] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.266175 26725 raft_consensus.cc:374] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.266050 26724 consensus_queue.cc:260] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.266302 26725 raft_consensus.cc:729] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.266666 26725 consensus_queue.cc:260] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.266881 26724 ts_tablet_manager.cc:1428] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:42:08.267316 26725 ts_tablet_manager.cc:1428] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:08.268868 26726 tablet_bootstrap.cc:492] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: No bootstrap required, opened a new log
I20260808 15:42:08.269055 26726 ts_tablet_manager.cc:1397] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:42:08.269960 26726 raft_consensus.cc:348] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.270253 26726 raft_consensus.cc:374] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.270447 26726 raft_consensus.cc:729] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.270746 26726 consensus_queue.cc:260] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.271684 26726 ts_tablet_manager.cc:1428] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:08.312332 26730 raft_consensus.cc:482] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:08.312552 26730 raft_consensus.cc:504] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.314061 26730 leader_election.cc:288] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:08.335307 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.314558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:36499 (local address 127.23.130.3:36879)
0808 15:42:08.327228 (+ 12670us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.327234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.327587 (+ 353us) server_negotiation.cc:407] Connection header received
0808 15:42:08.327809 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.327817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.327906 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.328027 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:08.328763 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.329930 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.332809 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.334051 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.334225 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.334302 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.334809 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.334963 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11779,"thread_start_us":11580,"threads_started":1}
I20260808 15:42:08.335301 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.314351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:54341 (local address 127.23.130.1:46463)
0808 15:42:08.328107 (+ 13756us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.328114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.328140 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:42:08.328212 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.328217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.328285 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.328365 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:08.329030 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.330145 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.332806 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.334046 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.334255 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.334319 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.334806 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.334958 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13592,"thread_start_us":13444,"threads_started":1}
I20260808 15:42:08.335419 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.314462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.23.130.2:36499)
0808 15:42:08.327446 (+ 12984us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:08.327470 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:08.327643 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:08.328067 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:08.328070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:08.328098 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:08.328547 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.328565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.330076 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.330079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.332621 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.332630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.334312 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.334318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.334651 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.334677 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:42:08.334759 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 15:42:08.334945 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12789,"mutex_wait_us":156,"thread_start_us":12495,"threads_started":1}
I20260808 15:42:08.335441 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.314210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46463 (local address 127.23.130.2:54341)
0808 15:42:08.326930 (+ 12720us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:08.326958 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:08.327124 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:08.328423 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:08.328428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:08.328452 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:08.328868 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.328878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.330265 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.330270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.332613 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.332628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.334319 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.334324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.334572 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.334661 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 15:42:08.334762 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:42:08.334945 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12515,"mutex_wait_us":120,"thread_start_us":12330,"threads_started":1}
I20260808 15:42:08.336237 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3906fe0805364dca8a2b41e4e35420d3" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d" is_pre_election: true
I20260808 15:42:08.336489 26227 raft_consensus.cc:2454] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 0.
I20260808 15:42:08.336642 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3906fe0805364dca8a2b41e4e35420d3" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8" is_pre_election: true
I20260808 15:42:08.336907 26545 raft_consensus.cc:2454] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 0.
I20260808 15:42:08.337039 26305 leader_election.cc:302] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:08.337492 26730 raft_consensus.cc:2781] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:08.337643 26730 raft_consensus.cc:482] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:08.337761 26730 raft_consensus.cc:3037] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.340258 26730 raft_consensus.cc:504] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.340903 26730 leader_election.cc:288] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 election: Requested vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:08.341363 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3906fe0805364dca8a2b41e4e35420d3" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8"
I20260808 15:42:08.341413 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3906fe0805364dca8a2b41e4e35420d3" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d"
I20260808 15:42:08.341569 26545 raft_consensus.cc:3037] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.341595 26227 raft_consensus.cc:3037] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.344524 26545 raft_consensus.cc:2454] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 1.
I20260808 15:42:08.344524 26227 raft_consensus.cc:2454] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 1.
I20260808 15:42:08.344980 26307 leader_election.cc:302] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f54f48ba6d84607b613fde5128904b8, 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:08.345326 26730 raft_consensus.cc:2781] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:08.345795 26730 raft_consensus.cc:686] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 1 LEADER]: Becoming Leader. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Running, Role: LEADER
I20260808 15:42:08.346155 26730 consensus_queue.cc:237] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.349632 26049 catalog_manager.cc:5495] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da376c883a2405f97218e7efd2f5756 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "9da376c883a2405f97218e7efd2f5756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:08.394698 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.388636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.23.130.2:41545)
0808 15:42:08.389227 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.389233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.389264 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:42:08.389356 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.389360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.389435 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.389624 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:08.390100 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.390967 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.392315 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.392760 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.392894 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.393382 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:08.394025 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:08.394191 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.394250 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.394327 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":314,"threads_started":1}
I20260808 15:42:08.410167 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.404303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:38473 (local address 127.23.130.3:36879)
0808 15:42:08.404519 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.404523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.405219 (+ 696us) server_negotiation.cc:407] Connection header received
0808 15:42:08.405468 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.405472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.405539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.405621 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:08.406182 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.407010 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.408416 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.408838 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.408934 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.409598 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:08.409701 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:08.409770 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.409904 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.409971 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 15:42:08.410318 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.404294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:36879 (local address 127.23.130.2:38473)
0808 15:42:08.405112 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:08.405129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:08.405301 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:08.405646 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:08.405686 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:08.405701 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:08.406023 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.406032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.407170 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.407174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.408287 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.408300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.409081 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.409091 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.409319 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.409404 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:08.409793 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:08.409802 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:42:08.409870 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 15:42:08.409973 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":464,"threads_started":1}
I20260808 15:42:08.419137 26545 raft_consensus.cc:1261] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Refusing update from remote peer 9da376c883a2405f97218e7efd2f5756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:08.419138 26227 raft_consensus.cc:1261] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Refusing update from remote peer 9da376c883a2405f97218e7efd2f5756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:08.419837 26730 consensus_queue.cc:1035] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:08.420164 26736 consensus_queue.cc:1035] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:08.456437 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.450185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:56201 (local address 127.23.130.2:41545)
0808 15:42:08.450367 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:08.450374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:08.450964 (+ 590us) server_negotiation.cc:407] Connection header received
0808 15:42:08.451149 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:08.451155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:08.451232 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:08.451318 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:08.451735 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.452604 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.454685 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.455103 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.455273 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.455690 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:08.455797 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:08.455878 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:08.456010 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 15:42:08.456102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:42:08.456529 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:08.450147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41545 (local address 127.23.130.3:56201)
0808 15:42:08.450848 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:08.450869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:08.451005 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:08.451330 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:08.451333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:08.451342 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:08.451590 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.451599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.452751 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.452757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.454501 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:08.454521 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:08.455278 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:08.455284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:08.455463 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:08.455513 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:08.455894 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:08.455905 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:42:08.455987 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 15:42:08.456137 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"spinlock_wait_cycles":5888,"thread_start_us":516,"threads_started":1}
I20260808 15:42:08.490880 26049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:08.491161 26049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:08.494138 26049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Sending DeleteTablet for 3 replicas of tablet 3906fe0805364dca8a2b41e4e35420d3
I20260808 15:42:08.495048 26204 tablet_service.cc:1514] Processing DeleteTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:33494
I20260808 15:42:08.495225 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:56750
I20260808 15:42:08.495389 26712 tablet_replica.cc:330] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: stopping tablet replica
I20260808 15:42:08.495486 26715 tablet_replica.cc:330] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: stopping tablet replica
I20260808 15:42:08.495604 26712 raft_consensus.cc:2229] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:08.495682 26715 raft_consensus.cc:2229] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:08.495813 26712 raft_consensus.cc:2258] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:08.495869 26715 raft_consensus.cc:2258] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:08.496316 26356 tablet_service.cc:1514] Processing DeleteTablet for tablet 3906fe0805364dca8a2b41e4e35420d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:08 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260808 15:42:08.496685 26714 tablet_replica.cc:330] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: stopping tablet replica
I20260808 15:42:08.496897 26714 raft_consensus.cc:2229] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:08.497011 26715 ts_tablet_manager.cc:1905] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.496845 26049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 15:42:08.497411 26714 raft_consensus.cc:2258] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 15:42:08.497498 26049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:08.498654 26712 ts_tablet_manager.cc:1905] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.500895 26714 ts_tablet_manager.cc:1905] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:08.504607 26715 ts_tablet_manager.cc:1918] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:08.505196 26715 log.cc:1196] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/3906fe0805364dca8a2b41e4e35420d3
I20260808 15:42:08.505359 26712 ts_tablet_manager.cc:1918] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:08.505604 26712 log.cc:1196] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/3906fe0805364dca8a2b41e4e35420d3
I20260808 15:42:08.505734 26715 ts_tablet_manager.cc:1939] T 3906fe0805364dca8a2b41e4e35420d3 P 1f54f48ba6d84607b613fde5128904b8: Deleting consensus metadata
I20260808 15:42:08.506256 26712 ts_tablet_manager.cc:1939] T 3906fe0805364dca8a2b41e4e35420d3 P 540cf6e419e341af90d64164d5e2f16d: Deleting consensus metadata
I20260808 15:42:08.506978 26035 catalog_manager.cc:4841] TS 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879): tablet 3906fe0805364dca8a2b41e4e35420d3 (table test-table [id=933564a584f6412b9af67bbe38f6121d]) successfully deleted
I20260808 15:42:08.507488 26033 catalog_manager.cc:4841] TS 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463): tablet 3906fe0805364dca8a2b41e4e35420d3 (table test-table [id=933564a584f6412b9af67bbe38f6121d]) successfully deleted
I20260808 15:42:08.509261 26204 tablet_service.cc:1467] Processing CreateTablet for tablet c19630d152354b649c301f0253cd5260 (DEFAULT_TABLE table=test-table [id=4f1011015f184cd193f5d0650dac284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.509738 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19630d152354b649c301f0253cd5260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.509749 26516 tablet_service.cc:1467] Processing CreateTablet for tablet c19630d152354b649c301f0253cd5260 (DEFAULT_TABLE table=test-table [id=4f1011015f184cd193f5d0650dac284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.510159 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19630d152354b649c301f0253cd5260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.511703 26714 ts_tablet_manager.cc:1918] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:08.511947 26714 log.cc:1196] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/3906fe0805364dca8a2b41e4e35420d3
I20260808 15:42:08.512425 26714 ts_tablet_manager.cc:1939] T 3906fe0805364dca8a2b41e4e35420d3 P 9da376c883a2405f97218e7efd2f5756: Deleting consensus metadata
I20260808 15:42:08.513862 26034 catalog_manager.cc:4841] TS 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545): tablet 3906fe0805364dca8a2b41e4e35420d3 (table test-table [id=933564a584f6412b9af67bbe38f6121d]) successfully deleted
I20260808 15:42:08.514046 26356 tablet_service.cc:1467] Processing CreateTablet for tablet c19630d152354b649c301f0253cd5260 (DEFAULT_TABLE table=test-table [id=4f1011015f184cd193f5d0650dac284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:08.514482 26356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19630d152354b649c301f0253cd5260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:08.516192 26725 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Bootstrap starting.
I20260808 15:42:08.517970 26725 tablet_bootstrap.cc:654] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.519999 26724 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Bootstrap starting.
I20260808 15:42:08.522420 26726 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Bootstrap starting.
I20260808 15:42:08.522660 26724 tablet_bootstrap.cc:654] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.523968 26725 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: No bootstrap required, opened a new log
I20260808 15:42:08.524194 26725 ts_tablet_manager.cc:1397] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 15:42:08.524209 26726 tablet_bootstrap.cc:654] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:08.525285 26725 raft_consensus.cc:348] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.525573 26724 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: No bootstrap required, opened a new log
I20260808 15:42:08.525655 26725 raft_consensus.cc:374] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.525745 26724 ts_tablet_manager.cc:1397] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260808 15:42:08.525746 26725 raft_consensus.cc:729] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf6e419e341af90d64164d5e2f16d, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.526028 26725 consensus_queue.cc:260] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.526523 26724 raft_consensus.cc:348] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.526831 26724 raft_consensus.cc:374] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.526986 26724 raft_consensus.cc:729] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f54f48ba6d84607b613fde5128904b8, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.527335 26724 consensus_queue.cc:260] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.527922 26724 ts_tablet_manager.cc:1428] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 15:42:08.528463 26725 ts_tablet_manager.cc:1428] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260808 15:42:08.530490 26726 tablet_bootstrap.cc:492] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: No bootstrap required, opened a new log
I20260808 15:42:08.530750 26726 ts_tablet_manager.cc:1397] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:42:08.531658 26726 raft_consensus.cc:348] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.531978 26726 raft_consensus.cc:374] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:08.532110 26726 raft_consensus.cc:729] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Initialized, Role: FOLLOWER
I20260808 15:42:08.532407 26726 consensus_queue.cc:260] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.532985 26726 ts_tablet_manager.cc:1428] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 15:42:08.720415 26730 raft_consensus.cc:482] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:08.720672 26730 raft_consensus.cc:504] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.721421 26730 leader_election.cc:288] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:08.722054 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19630d152354b649c301f0253cd5260" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d" is_pre_election: true
I20260808 15:42:08.722311 26227 raft_consensus.cc:2454] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 0.
I20260808 15:42:08.722265 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19630d152354b649c301f0253cd5260" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8" is_pre_election: true
I20260808 15:42:08.722599 26545 raft_consensus.cc:2454] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 0.
I20260808 15:42:08.722802 26305 leader_election.cc:302] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:08.723150 26730 raft_consensus.cc:2781] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:08.723285 26730 raft_consensus.cc:482] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:08.723402 26730 raft_consensus.cc:3037] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.726804 26730 raft_consensus.cc:504] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.727582 26730 leader_election.cc:288] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 election: Requested vote from peers 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463), 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879)
I20260808 15:42:08.728250 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19630d152354b649c301f0253cd5260" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540cf6e419e341af90d64164d5e2f16d"
I20260808 15:42:08.728345 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19630d152354b649c301f0253cd5260" candidate_uuid: "9da376c883a2405f97218e7efd2f5756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f54f48ba6d84607b613fde5128904b8"
I20260808 15:42:08.728497 26227 raft_consensus.cc:3037] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.728636 26545 raft_consensus.cc:3037] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:08.731204 26227 raft_consensus.cc:2454] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 1.
I20260808 15:42:08.731763 26305 leader_election.cc:302] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540cf6e419e341af90d64164d5e2f16d, 9da376c883a2405f97218e7efd2f5756; no voters:
I20260808 15:42:08.731841 26545 raft_consensus.cc:2454] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376c883a2405f97218e7efd2f5756 in term 1.
I20260808 15:42:08.732218 26730 raft_consensus.cc:2781] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:08.732441 26730 raft_consensus.cc:686] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 1 LEADER]: Becoming Leader. State: Replica: 9da376c883a2405f97218e7efd2f5756, State: Running, Role: LEADER
I20260808 15:42:08.732756 26730 consensus_queue.cc:237] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } }
I20260808 15:42:08.735886 26049 catalog_manager.cc:5495] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da376c883a2405f97218e7efd2f5756 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "9da376c883a2405f97218e7efd2f5756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da376c883a2405f97218e7efd2f5756" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:08.782226 26545 raft_consensus.cc:1261] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Refusing update from remote peer 9da376c883a2405f97218e7efd2f5756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:08.782863 26227 raft_consensus.cc:1261] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Refusing update from remote peer 9da376c883a2405f97218e7efd2f5756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:08.782900 26730 consensus_queue.cc:1035] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f54f48ba6d84607b613fde5128904b8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:08.783430 26736 consensus_queue.cc:1035] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540cf6e419e341af90d64164d5e2f16d" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:42:08.825315 26692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:42:10.414841 26049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:10.415094 26049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:10.419159 26049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08662419dc4a47c78582741dd09ad919: Sending DeleteTablet for 3 replicas of tablet c19630d152354b649c301f0253cd5260
I20260808 15:42:10.420017 26204 tablet_service.cc:1514] Processing DeleteTablet for tablet c19630d152354b649c301f0253cd5260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:10 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:33494
I20260808 15:42:10.420020 26356 tablet_service.cc:1514] Processing DeleteTablet for tablet c19630d152354b649c301f0253cd5260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:10 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260808 15:42:10.420081 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet c19630d152354b649c301f0253cd5260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:10 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:56750
I20260808 15:42:10.420745 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26127
I20260808 15:42:10.425084 26782 tablet_replica.cc:330] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: stopping tablet replica
I20260808 15:42:10.425424 26782 raft_consensus.cc:2229] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:10.425827 26782 raft_consensus.cc:2258] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:10.426559 26781 tablet_replica.cc:330] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: stopping tablet replica
I20260808 15:42:10.426919 26781 raft_consensus.cc:2229] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:10.427277 26781 raft_consensus.cc:2258] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:10.428092 26782 ts_tablet_manager.cc:1905] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:10.428666 26781 ts_tablet_manager.cc:1905] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:10.431083 26780 tablet_replica.cc:330] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: stopping tablet replica
I20260808 15:42:10.431315 26780 raft_consensus.cc:2229] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:10.431864 26780 raft_consensus.cc:2258] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:10.432917 26780 ts_tablet_manager.cc:1905] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:10.436703 26782 ts_tablet_manager.cc:1918] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:10.436740 26781 ts_tablet_manager.cc:1918] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:10.436923 26782 log.cc:1196] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/c19630d152354b649c301f0253cd5260
I20260808 15:42:10.437055 26781 log.cc:1196] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/c19630d152354b649c301f0253cd5260
I20260808 15:42:10.437614 26782 ts_tablet_manager.cc:1939] T c19630d152354b649c301f0253cd5260 P 1f54f48ba6d84607b613fde5128904b8: Deleting consensus metadata
I20260808 15:42:10.437624 26781 ts_tablet_manager.cc:1939] T c19630d152354b649c301f0253cd5260 P 540cf6e419e341af90d64164d5e2f16d: Deleting consensus metadata
I20260808 15:42:10.438930 26035 catalog_manager.cc:4841] TS 1f54f48ba6d84607b613fde5128904b8 (127.23.130.3:36879): tablet c19630d152354b649c301f0253cd5260 (table test-table [id=4f1011015f184cd193f5d0650dac284a]) successfully deleted
I20260808 15:42:10.439250 26033 catalog_manager.cc:4841] TS 540cf6e419e341af90d64164d5e2f16d (127.23.130.1:46463): tablet c19630d152354b649c301f0253cd5260 (table test-table [id=4f1011015f184cd193f5d0650dac284a]) successfully deleted
I20260808 15:42:10.439889 26780 ts_tablet_manager.cc:1918] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:42:10.440035 26780 log.cc:1196] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/c19630d152354b649c301f0253cd5260
I20260808 15:42:10.440569 26780 ts_tablet_manager.cc:1939] T c19630d152354b649c301f0253cd5260 P 9da376c883a2405f97218e7efd2f5756: Deleting consensus metadata
I20260808 15:42:10.441555 26034 catalog_manager.cc:4841] TS 9da376c883a2405f97218e7efd2f5756 (127.23.130.2:41545): tablet c19630d152354b649c301f0253cd5260 (table test-table [id=4f1011015f184cd193f5d0650dac284a]) successfully deleted
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:10.455581 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:10.422213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.23.130.1:46463)
0808 15:42:10.422905 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:10.422914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:10.431158 (+ 8244us) server_negotiation.cc:407] Connection header received
0808 15:42:10.431397 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:10.431403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:10.431487 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:10.431577 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:10.433490 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.434715 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.436626 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.437334 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.437471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:10.448293 (+ 10822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:10.448312 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:10.448329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:10.448368 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:10.452269 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.453076 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.453085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.453087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.453253 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.453683 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.453689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.453693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.454127 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:10.454578 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:10.454802 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 15:42:10.455077 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":345,"threads_started":1}
W20260808 15:42:10.456362 26278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33512
W20260808 15:42:10.457044 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26127: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:10.457283 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26127
I20260808 15:42:10.484336 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26287
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:10.501088 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:10.485139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.23.130.2:41545)
0808 15:42:10.485726 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:10.485730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:10.485747 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:10.485805 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:10.485809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:10.485871 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:10.485985 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:10.487558 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.488431 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.489864 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.490309 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.490488 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:10.495236 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:10.495252 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:10.495268 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:10.495341 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:10.498294 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.499084 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.499091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.499094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.499231 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.499652 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.499654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.499656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.499995 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:10.500226 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:10.500538 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 15:42:10.500700 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":352,"threads_started":1}
W20260808 15:42:10.501870 26431 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58424
W20260808 15:42:10.502604 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26287: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:10.502808 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26287
I20260808 15:42:10.531734 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26441
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:10.547971 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:10.532572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.23.130.3:36879)
0808 15:42:10.533101 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:10.533105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:10.533122 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:10.533177 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:10.533180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:10.533244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:10.533333 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:10.535119 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.535881 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.537149 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:10.537576 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:10.537679 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:10.542324 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:10.542337 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:10.542383 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:10.542469 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:10.545180 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.545786 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.545793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.545795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.545936 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:10.546274 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:10.546277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:10.546279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:10.546754 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:10.546985 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:10.547310 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 15:42:10.547585 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":279,"threads_started":1}
W20260808 15:42:10.548731 26594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56778
W20260808 15:42:10.549463 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26441: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:10.549638 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26441
I20260808 15:42:10.575300 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26012
I20260808 15:42:10.576501 26094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:10.693156 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26012
2026-08-08T15:42:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10152 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[26790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[26790](info): set up 2 sockets
Aug 08 15:42:10 dist-test-slave-gp6n krb5kdc[26790](info): commencing operation
krb5kdc: starting...
W20260808 15:42:11.848321 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:11 dist-test-slave-gp6n krb5kdc[26790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:12.069988 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34147
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:12.409895 26803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:12.410220 26803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:12.410316 26803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:12.417778 26803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:12.417903 26803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:12.417951 26803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:12.418002 26803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:12.418049 26803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:12.428224 26803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:12.429697 26803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:12.431011 26803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:12.443951 26808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:12.443861 26809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:12.443897 26811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:12.444659 26803 server_base.cc:1034] running on GCE node
I20260808 15:42:12.445391 26803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:12.446802 26803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:12.448014 26803 hybrid_clock.cc:648] HybridClock initialized: now 1786203732447967 us; error 32 us; skew 500 ppm
I20260808 15:42:12.448380 26803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:12 dist-test-slave-gp6n krb5kdc[26790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:12.451792 26803 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:12.453464 26803 webserver.cc:460] Webserver started at http://127.23.130.62:37453/ using document root <none> and password file <none>
I20260808 15:42:12.454175 26803 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:12.454329 26803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:12.454674 26803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:12.457657 26803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "e6febc4ba2c74dfd8d0096571bcd1369"
format_stamp: "Formatted at 2026-08-08 15:42:12 on dist-test-slave-gp6n"
server_key: "82acbc8bf96bb7b56d21e95406df29e9"
server_key_iv: "6236f1c37befa99c94230672ce833696"
server_key_version: "encryptionkey@0"
I20260808 15:42:12.458510 26803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "e6febc4ba2c74dfd8d0096571bcd1369"
format_stamp: "Formatted at 2026-08-08 15:42:12 on dist-test-slave-gp6n"
server_key: "82acbc8bf96bb7b56d21e95406df29e9"
server_key_iv: "6236f1c37befa99c94230672ce833696"
server_key_version: "encryptionkey@0"
I20260808 15:42:12.463765 26803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:12.467218 26818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:12.468576 26803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:12.468768 26803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "e6febc4ba2c74dfd8d0096571bcd1369"
format_stamp: "Formatted at 2026-08-08 15:42:12 on dist-test-slave-gp6n"
server_key: "82acbc8bf96bb7b56d21e95406df29e9"
server_key_iv: "6236f1c37befa99c94230672ce833696"
server_key_version: "encryptionkey@0"
I20260808 15:42:12.468981 26803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:12.496127 26803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:12.500602 26803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:12.500908 26803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:12.519639 26803 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:34147
I20260808 15:42:12.519681 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:34147 every 8 connection(s)
I20260808 15:42:12.521049 26803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:12.521593 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26803
I20260808 15:42:12.521864 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:12.522735 24072 external_mini_cluster.cc:1428] Setting key a88696a1d3419d9f470bc37e2cf503c3
I20260808 15:42:12.526019 26872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:42:12 dist-test-slave-gp6n krb5kdc[26790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:12.540156 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369: Bootstrap starting.
I20260808 15:42:12.544509 26872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:12.546026 26872 log.cc:824] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:12.549705 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369: No bootstrap required, opened a new log
I20260808 15:42:12.550851 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:12.524774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.23.130.62:34147)
0808 15:42:12.525701 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:12.525717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:12.525772 (+ 55us) server_negotiation.cc:407] Connection header received
0808 15:42:12.528426 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:12.528454 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:12.528957 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:12.529500 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:12.531204 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:12.533568 (+ 2364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:12.535117 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:12.536274 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:12.536481 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:12.541195 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:12.541207 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:12.541225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:12.541273 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:12.545798 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:12.546727 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:12.546737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:12.546746 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:12.546959 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:12.547461 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:12.547466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:12.547469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:12.548234 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:12.548570 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:12.548896 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 15:42:12.549390 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":357,"threads_started":1}
I20260808 15:42:12.558773 26872 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } }
I20260808 15:42:12.559145 26872 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:12.559276 26872 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6febc4ba2c74dfd8d0096571bcd1369, State: Initialized, Role: FOLLOWER
I20260808 15:42:12.559975 26872 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } }
I20260808 15:42:12.560293 26872 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:12.560464 26872 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:12.560667 26872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:12.563097 26872 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } }
I20260808 15:42:12.563621 26872 leader_election.cc:302] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6febc4ba2c74dfd8d0096571bcd1369; no voters:
I20260808 15:42:12.564234 26872 leader_election.cc:288] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:12.564432 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:12.565266 26878 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [term 1 LEADER]: Becoming Leader. State: Replica: e6febc4ba2c74dfd8d0096571bcd1369, State: Running, Role: LEADER
I20260808 15:42:12.565840 26878 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } }
I20260808 15:42:12.566711 26872 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:12.568684 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } } }
I20260808 15:42:12.569017 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:12.569908 26885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:12.569723 26881 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6febc4ba2c74dfd8d0096571bcd1369. Latest consensus state: current_term: 1 leader_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6febc4ba2c74dfd8d0096571bcd1369" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34147 } } }
I20260808 15:42:12.570070 26881 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:12.576668 26885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:12.585709 26885 catalog_manager.cc:1347] Generated new cluster ID: eab2209155d34ce9a87cf88a5883668b
I20260808 15:42:12.585819 26885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:12.597500 26885 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:12.599028 26885 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:12.605304 26885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6febc4ba2c74dfd8d0096571bcd1369: Generated new TSK 0
I20260808 15:42:12.606078 26885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:12.666867 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:13.000584 26903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:13.000874 26903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:13.001056 26903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:13.008561 26903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:13.008688 26903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:13.008903 26903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:13.018893 26903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:13.020265 26903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:13.021574 26903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:13.035198 26908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:13.035166 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:13.035130 26909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:13.035919 26903 server_base.cc:1034] running on GCE node
I20260808 15:42:13.036489 26903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:13.037379 26903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:13.038592 26903 hybrid_clock.cc:648] HybridClock initialized: now 1786203733038538 us; error 44 us; skew 500 ppm
I20260808 15:42:13.038949 26903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:13 dist-test-slave-gp6n krb5kdc[26790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:13.042182 26903 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:13.043781 26903 webserver.cc:460] Webserver started at http://127.23.130.1:43371/ using document root <none> and password file <none>
I20260808 15:42:13.044510 26903 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:13.044668 26903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:13.044999 26903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:13.048036 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "91ead5ed4e85441ba13fba6cc1dc8ab9"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "c53db85c3b3d68b5b1311445e3c0764d"
server_key_iv: "601f1a27b7eecc1e6e183b300fcaa131"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.048844 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "91ead5ed4e85441ba13fba6cc1dc8ab9"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "c53db85c3b3d68b5b1311445e3c0764d"
server_key_iv: "601f1a27b7eecc1e6e183b300fcaa131"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.054281 26903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:42:13.057654 26919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.059060 26903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:13.059248 26903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "91ead5ed4e85441ba13fba6cc1dc8ab9"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "c53db85c3b3d68b5b1311445e3c0764d"
server_key_iv: "601f1a27b7eecc1e6e183b300fcaa131"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.059490 26903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:13.098999 26903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:13.102949 26903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:13.103259 26903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:13.104213 26903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:13.105671 26903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:13.105780 26903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.105944 26903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:13.106017 26903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.140723 26903 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:42469
I20260808 15:42:13.140789 27047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:42469 every 8 connection(s)
I20260808 15:42:13.142109 26903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:13.151558 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 26903
I20260808 15:42:13.151765 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:13.152405 24072 external_mini_cluster.cc:1428] Setting key ef1792761117429f9b1b3e6fc9ea5c67
Aug 08 15:42:13 dist-test-slave-gp6n krb5kdc[26790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:13.163914 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:13.144783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:58073 (local address 127.23.130.62:34147)
0808 15:42:13.145206 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:13.145210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:13.146228 (+ 1018us) server_negotiation.cc:407] Connection header received
0808 15:42:13.148226 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:13.148234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:13.148330 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:13.148460 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:13.150021 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.150806 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.152496 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.152904 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.153063 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:13.157002 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:13.157011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:13.157018 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:13.157061 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:13.160599 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:13.161314 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:13.161321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:13.161327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:13.161475 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:13.161881 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:13.161889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:13.161895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:13.162242 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:13.162354 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:13.163336 (+ 982us) server_negotiation.cc:299] Negotiation successful
0808 15:42:13.163583 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260808 15:42:13.165427 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:13.145364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34147 (local address 127.23.130.1:58073)
0808 15:42:13.146066 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:13.146105 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:13.147926 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:13.148711 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:13.148719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:13.149256 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:13.149847 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:13.149865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.150982 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.150987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:13.152350 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:13.152363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.153154 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.153160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:13.153491 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:13.154177 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:13.154199 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:13.156773 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:13.160748 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:13.160754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:13.160767 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:13.161198 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:13.161582 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:13.161585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:13.161588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:13.161772 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:13.162375 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:13.162386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:13.162883 (+ 497us) client_negotiation.cc:769] Sending connection context
0808 15:42:13.163242 (+ 359us) client_negotiation.cc:240] Negotiation successful
0808 15:42:13.163821 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260808 15:42:13.166970 27048 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34147
I20260808 15:42:13.167277 27048 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:13.168109 27048 heartbeater.cc:511] Master 127.23.130.62:34147 requested a full tablet report, sending...
I20260808 15:42:13.170149 26835 ts_manager.cc:194] Registered new tserver with Master: 91ead5ed4e85441ba13fba6cc1dc8ab9 (127.23.130.1:42469)
I20260808 15:42:13.171756 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:58073
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:13.205384 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:13.545044 27055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:13.545307 27055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:13.545401 27055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:13.552722 27055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:13.552835 27055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:13.552968 27055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:13.562521 27055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:13.563807 27055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:13.565126 27055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:13.578200 27061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:13.578223 27063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:13.578346 27060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:13.579267 27055 server_base.cc:1034] running on GCE node
I20260808 15:42:13.579800 27055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:13.580672 27055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:13.581876 27055 hybrid_clock.cc:648] HybridClock initialized: now 1786203733581840 us; error 33 us; skew 500 ppm
I20260808 15:42:13.582151 27055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:13 dist-test-slave-gp6n krb5kdc[26790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:13.585222 27055 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:13.586791 27055 webserver.cc:460] Webserver started at http://127.23.130.2:38849/ using document root <none> and password file <none>
I20260808 15:42:13.587531 27055 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:13.587661 27055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:13.587983 27055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:13.590894 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "26ab48a188c14c79b17eacf63d6a5c51"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "23ac7a170178b5e5e5a83b8afbd9ba07"
server_key_iv: "bb3b1b687b7e74484b49cefa72560252"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.591666 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "26ab48a188c14c79b17eacf63d6a5c51"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "23ac7a170178b5e5e5a83b8afbd9ba07"
server_key_iv: "bb3b1b687b7e74484b49cefa72560252"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.596680 27055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:13.600039 27071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.601326 27055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:13.601521 27055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "26ab48a188c14c79b17eacf63d6a5c51"
format_stamp: "Formatted at 2026-08-08 15:42:13 on dist-test-slave-gp6n"
server_key: "23ac7a170178b5e5e5a83b8afbd9ba07"
server_key_iv: "bb3b1b687b7e74484b49cefa72560252"
server_key_version: "encryptionkey@0"
I20260808 15:42:13.601784 27055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:13.629921 27055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:13.634261 27055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:13.634606 27055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:13.635635 27055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:13.636955 27055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:13.637059 27055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.637205 27055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:13.637279 27055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:13.672290 27055 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:35295
I20260808 15:42:13.672379 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:35295 every 8 connection(s)
I20260808 15:42:13.673661 27055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:13.675827 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27055
I20260808 15:42:13.676023 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:13.676627 24072 external_mini_cluster.cc:1428] Setting key 0986503d2b529fcfcf8211a0d1f3902d
Aug 08 15:42:13 dist-test-slave-gp6n krb5kdc[26790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:13.693813 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:13.676323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:33567 (local address 127.23.130.62:34147)
0808 15:42:13.676795 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:13.676800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:13.678067 (+ 1267us) server_negotiation.cc:407] Connection header received
0808 15:42:13.680149 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:13.680155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:13.680242 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:13.680362 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:13.681871 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.682657 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.684184 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.684625 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.684760 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:13.688196 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:13.688200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:13.688204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:13.688235 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:13.690843 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:13.691539 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:13.691544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:13.691548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:13.691677 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:13.692101 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:13.692109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:13.692113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:13.692395 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:13.692552 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:13.693205 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 15:42:13.693428 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":245,"threads_started":1}
I20260808 15:42:13.694811 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:13.677010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34147 (local address 127.23.130.2:33567)
0808 15:42:13.677880 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:13.677930 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:13.679865 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:13.680608 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:13.680616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:13.681141 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:13.681700 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:13.681716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.682796 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.682803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:13.684039 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:13.684051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:13.684860 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:13.684868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:13.685110 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:13.685795 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:13.685816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:13.687966 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:13.690978 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:13.690994 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:13.691008 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:13.691416 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:13.691785 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:13.691791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:13.691795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:13.691982 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:13.692533 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:13.692540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:13.692856 (+ 316us) client_negotiation.cc:769] Sending connection context
0808 15:42:13.693125 (+ 269us) client_negotiation.cc:240] Negotiation successful
0808 15:42:13.693626 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":354,"threads_started":1}
I20260808 15:42:13.696290 27199 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34147
I20260808 15:42:13.696605 27199 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:42:13.697335 27199 heartbeater.cc:511] Master 127.23.130.62:34147 requested a full tablet report, sending...
I20260808 15:42:13.698799 26835 ts_manager.cc:194] Registered new tserver with Master: 26ab48a188c14c79b17eacf63d6a5c51 (127.23.130.2:35295)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
I20260808 15:42:13.699724 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:33567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:13.725677 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:14.059273 27206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:14.059561 27206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:14.059656 27206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:14.066985 27206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:14.067112 27206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:14.067261 27206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:14.077162 27206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:14.078387 27206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:14.079751 27206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:14.092916 27214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:14.092830 27212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:14.092950 27211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:14.093844 27206 server_base.cc:1034] running on GCE node
I20260808 15:42:14.094504 27206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:14.095418 27206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:14.096653 27206 hybrid_clock.cc:648] HybridClock initialized: now 1786203734096613 us; error 50 us; skew 500 ppm
I20260808 15:42:14.096974 27206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:14 dist-test-slave-gp6n krb5kdc[26790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203734, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:14.100265 27206 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:14.101689 27206 webserver.cc:460] Webserver started at http://127.23.130.3:37811/ using document root <none> and password file <none>
I20260808 15:42:14.102384 27206 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:14.102603 27206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:14.102924 27206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:14.105957 27206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "c7942a091375465db63a94b027b4ebdf"
format_stamp: "Formatted at 2026-08-08 15:42:14 on dist-test-slave-gp6n"
server_key: "cbc1c82c1387995e8906476d17d710a4"
server_key_iv: "8e223f449b0ab9b54ec1d016672711d0"
server_key_version: "encryptionkey@0"
I20260808 15:42:14.106825 27206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "c7942a091375465db63a94b027b4ebdf"
format_stamp: "Formatted at 2026-08-08 15:42:14 on dist-test-slave-gp6n"
server_key: "cbc1c82c1387995e8906476d17d710a4"
server_key_iv: "8e223f449b0ab9b54ec1d016672711d0"
server_key_version: "encryptionkey@0"
I20260808 15:42:14.111764 27206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:14.115125 27221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:14.116372 27206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:14.116575 27206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "c7942a091375465db63a94b027b4ebdf"
format_stamp: "Formatted at 2026-08-08 15:42:14 on dist-test-slave-gp6n"
server_key: "cbc1c82c1387995e8906476d17d710a4"
server_key_iv: "8e223f449b0ab9b54ec1d016672711d0"
server_key_version: "encryptionkey@0"
I20260808 15:42:14.116787 27206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:14.144560 27206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:14.148871 27206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:14.149168 27206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:14.150130 27206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:14.151491 27206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:14.151598 27206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:14.151744 27206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:14.151820 27206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:14.175170 27048 heartbeater.cc:503] Master 127.23.130.62:34147 was elected leader, sending a full tablet report...
I20260808 15:42:14.185825 27206 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:40425
I20260808 15:42:14.185940 27346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:40425 every 8 connection(s)
I20260808 15:42:14.187213 27206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:14.196273 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27206
I20260808 15:42:14.196447 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:14.197007 24072 external_mini_cluster.cc:1428] Setting key e1ebe20639adb374a32c6d473dfd3a8e
Aug 08 15:42:14 dist-test-slave-gp6n krb5kdc[26790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203734, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:14.207062 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:14.189708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:51919 (local address 127.23.130.62:34147)
0808 15:42:14.189929 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:14.189936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:14.191125 (+ 1189us) server_negotiation.cc:407] Connection header received
0808 15:42:14.192850 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:14.192855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:14.192923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:14.193020 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:14.194458 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:14.195210 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:14.197265 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:14.197654 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:14.197794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:14.201170 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:14.201175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:14.201178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:14.201210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:14.203922 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:14.204650 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:14.204657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:14.204661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:14.204788 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:14.205285 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:14.205290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:14.205293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:14.205589 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:14.205744 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:14.206492 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 15:42:14.206721 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:42:14.207964 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:14.190245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34147 (local address 127.23.130.3:51919)
0808 15:42:14.190970 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:14.191014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:14.192611 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:14.193218 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:14.193226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:14.193717 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:14.194270 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:14.194287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:14.195343 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:14.195347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:14.197124 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:14.197139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:14.197879 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:14.197886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:14.198118 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:14.198819 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:14.198841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:14.200967 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:14.204078 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:14.204086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:14.204101 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:14.204528 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:14.204903 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:14.204908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:14.204912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:14.205161 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:14.205719 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:14.205727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:14.206054 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 15:42:14.206365 (+ 311us) client_negotiation.cc:240] Negotiation successful
0808 15:42:14.206909 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":319,"threads_started":1}
I20260808 15:42:14.209208 27347 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34147
I20260808 15:42:14.209519 27347 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:14.210209 27347 heartbeater.cc:511] Master 127.23.130.62:34147 requested a full tablet report, sending...
I20260808 15:42:14.211771 26835 ts_manager.cc:194] Registered new tserver with Master: c7942a091375465db63a94b027b4ebdf (127.23.130.3:40425)
I20260808 15:42:14.212602 26835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:51919
I20260808 15:42:14.224761 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:42:14.260124 27202 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:14.257447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.23.130.62:34147)
0808 15:42:14.257696 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:14.257705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:14.258070 (+ 365us) server_negotiation.cc:407] Connection header received
0808 15:42:14.258287 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:14.258291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:14.258366 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:14.258511 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:14.259801 (+ 1290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33848: BlockingRecv error: recv got EOF from 127.0.0.1:33848 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:42:14.263273 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26903
W20260808 15:42:14.266096 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26903: Not authorized: Client connection negotiation failed: client connection to 127.23.130.1:42469: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:42:14.266299 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26903
I20260808 15:42:14.286350 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27055
W20260808 15:42:14.289026 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27055: Not authorized: Client connection negotiation failed: client connection to 127.23.130.2:35295: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:42:14.289186 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27055
I20260808 15:42:14.307350 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27206
W20260808 15:42:14.310474 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27206: Not authorized: Client connection negotiation failed: client connection to 127.23.130.3:40425: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:42:14.310654 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27206
I20260808 15:42:14.326002 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 26803
I20260808 15:42:14.327487 26867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:14.430433 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 26803
2026-08-08T15:42:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3733 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:14 dist-test-slave-gp6n krb5kdc[27366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:14 dist-test-slave-gp6n krb5kdc[27366](info): set up 2 sockets
Aug 08 15:42:14 dist-test-slave-gp6n krb5kdc[27366](info): commencing operation
krb5kdc: starting...
W20260808 15:42:15.595655 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:15 dist-test-slave-gp6n krb5kdc[27366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:15.809625 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38129
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:16.132681 27379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:16.132968 27379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:16.133153 27379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:16.141764 27379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:16.141865 27379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:16.141968 27379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:16.142052 27379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:16.142125 27379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:16.152102 27379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:16.153462 27379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:16.154862 27379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:16.166890 27384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:16.166961 27386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:16.166991 27388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:16.168089 27379 server_base.cc:1034] running on GCE node
I20260808 15:42:16.168962 27379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:16.170439 27379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:16.171684 27379 hybrid_clock.cc:648] HybridClock initialized: now 1786203736171636 us; error 43 us; skew 500 ppm
I20260808 15:42:16.172061 27379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:16 dist-test-slave-gp6n krb5kdc[27366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203736, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:16.180614 27379 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:16.182389 27379 webserver.cc:460] Webserver started at http://127.23.130.62:43177/ using document root <none> and password file <none>
I20260808 15:42:16.183243 27379 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:16.183409 27379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:16.183763 27379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:16.187095 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "60cc8a591dbe454792b8ca53c3c157cb"
format_stamp: "Formatted at 2026-08-08 15:42:16 on dist-test-slave-gp6n"
server_key: "22f40e06eb4be92a49496b7a1559910d"
server_key_iv: "fd580a277c56be3d5e86b89bf0d33c89"
server_key_version: "encryptionkey@0"
I20260808 15:42:16.187963 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "60cc8a591dbe454792b8ca53c3c157cb"
format_stamp: "Formatted at 2026-08-08 15:42:16 on dist-test-slave-gp6n"
server_key: "22f40e06eb4be92a49496b7a1559910d"
server_key_iv: "fd580a277c56be3d5e86b89bf0d33c89"
server_key_version: "encryptionkey@0"
I20260808 15:42:16.193392 27379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:42:16.196990 27395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:16.198293 27379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:16.198544 27379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "60cc8a591dbe454792b8ca53c3c157cb"
format_stamp: "Formatted at 2026-08-08 15:42:16 on dist-test-slave-gp6n"
server_key: "22f40e06eb4be92a49496b7a1559910d"
server_key_iv: "fd580a277c56be3d5e86b89bf0d33c89"
server_key_version: "encryptionkey@0"
I20260808 15:42:16.198753 27379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:16.218873 27379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:16.225174 27379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:16.225541 27379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:16.246757 27379 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:38129
I20260808 15:42:16.246767 27451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:38129 every 8 connection(s)
I20260808 15:42:16.248204 27379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:16.250751 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27379
I20260808 15:42:16.251040 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:16.251859 24072 external_mini_cluster.cc:1428] Setting key 08de242cc161c300636341503f73bb27
I20260808 15:42:16.252748 27452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:42:16 dist-test-slave-gp6n krb5kdc[27366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:16.267269 27452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb: Bootstrap starting.
I20260808 15:42:16.270443 27452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:16.271500 27452 log.cc:824] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:16.274317 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:16.253767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.23.130.62:38129)
0808 15:42:16.254564 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:16.254576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:16.254614 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:42:16.256137 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:16.256157 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:16.256521 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:16.256844 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:16.258661 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:16.259964 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:16.261752 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:16.262379 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:16.262574 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:16.267205 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:16.267213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:16.267226 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:16.267267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:16.270271 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:16.271043 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:16.271051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:16.271057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:16.271240 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:16.271640 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:16.271644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:16.271646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:16.272352 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:16.272671 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:16.272980 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 15:42:16.273345 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":383,"threads_started":1}
I20260808 15:42:16.274606 27452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb: No bootstrap required, opened a new log
I20260808 15:42:16.278185 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:42:16.282401 27452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } }
I20260808 15:42:16.282830 27452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:16.282938 27452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cc8a591dbe454792b8ca53c3c157cb, State: Initialized, Role: FOLLOWER
I20260808 15:42:16.283491 27452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } }
I20260808 15:42:16.283740 27452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:16.283886 27452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:16.284067 27452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:16.286160 27452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } }
I20260808 15:42:16.286741 27452 leader_election.cc:302] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60cc8a591dbe454792b8ca53c3c157cb; no voters:
I20260808 15:42:16.287385 27452 leader_election.cc:288] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:16.287483 27458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:16.289955 27452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:16.290680 27458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [term 1 LEADER]: Becoming Leader. State: Replica: 60cc8a591dbe454792b8ca53c3c157cb, State: Running, Role: LEADER
I20260808 15:42:16.291325 27458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } }
I20260808 15:42:16.294787 27460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60cc8a591dbe454792b8ca53c3c157cb. Latest consensus state: current_term: 1 leader_uuid: "60cc8a591dbe454792b8ca53c3c157cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } } }
I20260808 15:42:16.295239 27460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:16.295976 27459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60cc8a591dbe454792b8ca53c3c157cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc8a591dbe454792b8ca53c3c157cb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38129 } } }
I20260808 15:42:16.296229 27469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:16.296271 27459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:16.304742 27469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:16.331449 27469 catalog_manager.cc:1347] Generated new cluster ID: eb42b455c0eb4d6ea931ae09f3c43db3
I20260808 15:42:16.331590 27469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:16.377395 27469 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:16.379392 27469 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:16.386356 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:16.372496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.23.130.62:38129)
0808 15:42:16.372688 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:16.372695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:16.376710 (+ 4015us) server_negotiation.cc:407] Connection header received
0808 15:42:16.376990 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:16.376997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:16.377086 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:16.377163 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:16.379467 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:16.380651 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:16.382540 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:16.383218 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:16.383349 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:16.384614 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:16.384620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:16.384625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:16.385461 (+ 836us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:16.385539 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:16.385580 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:16.385684 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:16.385841 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 15:42:16.385979 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:42:16.391636 27469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60cc8a591dbe454792b8ca53c3c157cb: Generated new TSK 0
I20260808 15:42:16.392656 27469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:42:16.403194 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27379
I20260808 15:42:16.404538 27446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:16.496734 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27379
2026-08-08T15:42:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2060 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:16 dist-test-slave-gp6n krb5kdc[27489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:16 dist-test-slave-gp6n krb5kdc[27489](info): set up 2 sockets
Aug 08 15:42:16 dist-test-slave-gp6n krb5kdc[27489](info): commencing operation
krb5kdc: starting...
W20260808 15:42:17.632759 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:17 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:17.849553 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44183
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:44183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:18.175139 27505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:18.175417 27505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:18.175624 27505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:18.182788 27505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:18.182891 27505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:18.182996 27505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:18.183099 27505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:18.183185 27505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:18.192795 27505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:44183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:18.194062 27505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:18.195379 27505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:18.208271 27514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:18.208359 27511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:18.208329 27512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:18.209532 27505 server_base.cc:1034] running on GCE node
I20260808 15:42:18.210304 27505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:18.211694 27505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:18.212929 27505 hybrid_clock.cc:648] HybridClock initialized: now 1786203738212892 us; error 53 us; skew 500 ppm
I20260808 15:42:18.213300 27505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:18 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203738, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:18.216802 27505 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:18.218724 27505 webserver.cc:460] Webserver started at http://127.23.130.62:35905/ using document root <none> and password file <none>
I20260808 15:42:18.219538 27505 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:18.219687 27505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:18.220017 27505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:18.223843 27505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "d17aaac0e13043d68891c80dbcc3a838"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "c11c061bc5120a7e6a40a91876ef7baf"
server_key_iv: "0a03fdda2f2ec9402a552fd874496627"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.224972 27505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "d17aaac0e13043d68891c80dbcc3a838"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "c11c061bc5120a7e6a40a91876ef7baf"
server_key_iv: "0a03fdda2f2ec9402a552fd874496627"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.232478 27505 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260808 15:42:18.236089 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:18.237465 27505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:18.237660 27505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "d17aaac0e13043d68891c80dbcc3a838"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "c11c061bc5120a7e6a40a91876ef7baf"
server_key_iv: "0a03fdda2f2ec9402a552fd874496627"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.237855 27505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:18.262835 27505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:18.267654 27505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:18.267975 27505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:18.288182 27505 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:44183
I20260808 15:42:18.288218 27577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:44183 every 8 connection(s)
I20260808 15:42:18.289650 27505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:18.293332 27578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:18.297897 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27505
I20260808 15:42:18.298157 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:18.299003 24072 external_mini_cluster.cc:1428] Setting key eb362c31ef382054406a83325cc55185
I20260808 15:42:18.304983 27578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838: Bootstrap starting.
I20260808 15:42:18.308594 27578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:18.309739 27578 log.cc:824] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:18.312400 27578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838: No bootstrap required, opened a new log
Aug 08 15:42:18 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:18.320670 27578 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } }
I20260808 15:42:18.321094 27578 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:18.321226 27578 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17aaac0e13043d68891c80dbcc3a838, State: Initialized, Role: FOLLOWER
I20260808 15:42:18.321893 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:18.300992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.23.130.62:44183)
0808 15:42:18.301747 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:18.301758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:18.301803 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:42:18.303584 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:18.303606 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:18.303949 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:18.304264 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:18.306027 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.307400 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.309174 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.309871 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.310030 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:18.314540 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:18.314548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:18.314565 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:18.314599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:18.317675 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:18.318427 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:18.318436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:18.318444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:18.318632 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:18.319043 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:18.319049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:18.319052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:18.319592 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:18.319942 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:18.320340 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 15:42:18.320763 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260808 15:42:18.321864 27578 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } }
I20260808 15:42:18.322165 27578 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:18.322319 27578 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:18.322542 27578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:18.325601 27578 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } }
I20260808 15:42:18.326099 27578 leader_election.cc:302] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17aaac0e13043d68891c80dbcc3a838; no voters:
I20260808 15:42:18.326749 27578 leader_election.cc:288] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:18.326895 27583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:18.328179 27583 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [term 1 LEADER]: Becoming Leader. State: Replica: d17aaac0e13043d68891c80dbcc3a838, State: Running, Role: LEADER
I20260808 15:42:18.328902 27578 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:18.328845 27583 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } }
I20260808 15:42:18.331806 27585 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17aaac0e13043d68891c80dbcc3a838. Latest consensus state: current_term: 1 leader_uuid: "d17aaac0e13043d68891c80dbcc3a838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } } }
I20260808 15:42:18.332190 27585 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:18.332183 27584 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17aaac0e13043d68891c80dbcc3a838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17aaac0e13043d68891c80dbcc3a838" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44183 } } }
I20260808 15:42:18.332528 27584 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:18.333223 27590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:18.339287 27590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:18.347620 27590 catalog_manager.cc:1347] Generated new cluster ID: 0c055079a51742878ad89a87b316c2dc
I20260808 15:42:18.347726 27590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:18.357566 27590 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:18.359364 27590 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:18.369491 27590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17aaac0e13043d68891c80dbcc3a838: Generated new TSK 0
I20260808 15:42:18.370513 27590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:18.435551 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:18.765461 27607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:18.765766 27607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:18.765969 27607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:18.773576 27607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:18.773696 27607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:18.773917 27607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:18.784101 27607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:18.785485 27607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:18.786957 27607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:18.800634 27613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:18.800611 27616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:18.800570 27614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:18.801396 27607 server_base.cc:1034] running on GCE node
I20260808 15:42:18.802054 27607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:18.803094 27607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:18.804330 27607 hybrid_clock.cc:648] HybridClock initialized: now 1786203738804276 us; error 69 us; skew 500 ppm
I20260808 15:42:18.804689 27607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:18 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203738, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:18.807945 27607 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:18.809397 27607 webserver.cc:460] Webserver started at http://127.23.130.1:41005/ using document root <none> and password file <none>
I20260808 15:42:18.810082 27607 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:18.810230 27607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:18.810588 27607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:18.813503 27607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "cb7d55a9afe94eee81893a9d4fd2c1de"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "fc7885e0081adaf06cc718f5bcb1aed8"
server_key_iv: "3e9dfea0636bf4efdcc0de1e29f35579"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.814296 27607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "cb7d55a9afe94eee81893a9d4fd2c1de"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "fc7885e0081adaf06cc718f5bcb1aed8"
server_key_iv: "3e9dfea0636bf4efdcc0de1e29f35579"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.819538 27607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:42:18.823485 27623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:18.824889 27607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:18.825070 27607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "cb7d55a9afe94eee81893a9d4fd2c1de"
format_stamp: "Formatted at 2026-08-08 15:42:18 on dist-test-slave-gp6n"
server_key: "fc7885e0081adaf06cc718f5bcb1aed8"
server_key_iv: "3e9dfea0636bf4efdcc0de1e29f35579"
server_key_version: "encryptionkey@0"
I20260808 15:42:18.825276 27607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:18.855919 27607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:18.860371 27607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:18.860741 27607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:18.861766 27607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:18.863160 27607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:18.863232 27607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:18.863377 27607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:18.863458 27607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:18.896399 27607 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:44671
I20260808 15:42:18.896440 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:44671 every 8 connection(s)
I20260808 15:42:18.897728 27607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:18.907788 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27607
I20260808 15:42:18.908064 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:18.908802 24072 external_mini_cluster.cc:1428] Setting key d652afca2230f0da46ed32df969b84f2
Aug 08 15:42:18 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203738, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:18.918550 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:18.900311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:48143 (local address 127.23.130.62:44183)
0808 15:42:18.900831 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:18.900838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:18.901815 (+ 977us) server_negotiation.cc:407] Connection header received
0808 15:42:18.903601 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:18.903606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:18.903698 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:18.903800 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:18.905365 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.906134 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.908070 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.908537 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.908643 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:18.912734 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:18.912739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:18.912743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:18.912775 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:18.915391 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:18.916163 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:18.916169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:18.916172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:18.916294 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:18.916740 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:18.916745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:18.916747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:18.917010 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:18.917170 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:18.917905 (+ 735us) server_negotiation.cc:299] Negotiation successful
0808 15:42:18.918141 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260808 15:42:18.919574 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:18.900930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44183 (local address 127.23.130.1:48143)
0808 15:42:18.901641 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:18.901680 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:18.903329 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:18.904010 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:18.904018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:18.904549 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:18.905135 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:18.905153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.906278 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.906281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:18.907855 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:18.907872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:18.908782 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:18.908791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:18.909111 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:18.909983 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:18.910015 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:18.912511 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:18.915561 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:18.915570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:18.915588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:18.916024 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:18.916409 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:18.916414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:18.916417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:18.916613 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:18.917144 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:18.917151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:18.917514 (+ 363us) client_negotiation.cc:769] Sending connection context
0808 15:42:18.917789 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 15:42:18.918359 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":294,"threads_started":1}
I20260808 15:42:18.921253 27744 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44183
I20260808 15:42:18.921591 27744 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:18.922359 27744 heartbeater.cc:511] Master 127.23.130.62:44183 requested a full tablet report, sending...
I20260808 15:42:18.924563 27541 ts_manager.cc:194] Registered new tserver with Master: cb7d55a9afe94eee81893a9d4fd2c1de (127.23.130.1:44671)
I20260808 15:42:18.926028 27541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:48143
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:18.959398 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:19.290198 27750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:19.290582 27750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:19.290812 27750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:19.298708 27750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:19.298810 27750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:19.299022 27750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:19.308851 27750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:19.310111 27750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:19.311852 27750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:19.324965 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:19.324972 27756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:19.325141 27761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:19.325800 27750 server_base.cc:1034] running on GCE node
I20260808 15:42:19.326453 27750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:19.327437 27750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:19.328601 27750 hybrid_clock.cc:648] HybridClock initialized: now 1786203739328544 us; error 52 us; skew 500 ppm
I20260808 15:42:19.329058 27750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:19.332571 27750 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:19.333945 27750 webserver.cc:460] Webserver started at http://127.23.130.2:33239/ using document root <none> and password file <none>
I20260808 15:42:19.334786 27750 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:19.334961 27750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:19.335403 27750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:19.338276 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "ee63e4d561194432afaddbd3d51c83fd"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "63395c93155f7f53addbcc8ea4e350a1"
server_key_iv: "93ff8467f86bedb0742ba628aa15764c"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.339118 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "ee63e4d561194432afaddbd3d51c83fd"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "63395c93155f7f53addbcc8ea4e350a1"
server_key_iv: "93ff8467f86bedb0742ba628aa15764c"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.344393 27750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:19.348044 27770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.349385 27750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:19.349570 27750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "ee63e4d561194432afaddbd3d51c83fd"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "63395c93155f7f53addbcc8ea4e350a1"
server_key_iv: "93ff8467f86bedb0742ba628aa15764c"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.349769 27750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:19.366927 27750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:19.371263 27750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:19.371569 27750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:19.372560 27750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:19.373903 27750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:19.373982 27750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.374150 27750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:19.374213 27750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.408274 27750 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41371
I20260808 15:42:19.408321 27888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41371 every 8 connection(s)
I20260808 15:42:19.409591 27750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:19.410678 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27750
I20260808 15:42:19.410888 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:19.411396 24072 external_mini_cluster.cc:1428] Setting key 491376b93f75557987f1e6a48ec97a8b
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:19.431398 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.412404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:39451 (local address 127.23.130.62:44183)
0808 15:42:19.412625 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:19.412631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:19.414024 (+ 1393us) server_negotiation.cc:407] Connection header received
0808 15:42:19.415892 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:19.415900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:19.416001 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:19.416112 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:19.417556 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.418705 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.420380 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.420907 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.421052 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:19.425065 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:19.425069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:19.425073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:19.425102 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:19.427772 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.428733 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.428737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.428740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.428899 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.429467 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.429473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.429477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.429858 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:19.430015 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:19.430820 (+ 805us) server_negotiation.cc:299] Negotiation successful
0808 15:42:19.431056 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:42:19.432399 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.412989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44183 (local address 127.23.130.2:39451)
0808 15:42:19.413829 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:19.413882 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:19.415624 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:19.416317 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:19.416325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:19.416817 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:19.417383 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:19.417399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.418852 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.418856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:19.420230 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:19.420249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.421152 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.421159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:19.421389 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:19.422065 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:19.422087 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:19.424844 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:19.427909 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:19.427920 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:19.427940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:19.428620 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:19.429040 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:19.429045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:19.429049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:19.429355 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:19.429975 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:19.429983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:19.430439 (+ 456us) client_negotiation.cc:769] Sending connection context
0808 15:42:19.430740 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 15:42:19.431243 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:42:19.433908 27889 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44183
I20260808 15:42:19.434297 27889 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:19.435145 27889 heartbeater.cc:511] Master 127.23.130.62:44183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
I20260808 15:42:19.436820 27541 ts_manager.cc:194] Registered new tserver with Master: ee63e4d561194432afaddbd3d51c83fd (127.23.130.2:41371)
I20260808 15:42:19.438156 27541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:39451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:19.462287 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:19.795763 27895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:19.796048 27895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:19.796145 27895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:19.803445 27895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:19.803555 27895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:19.803663 27895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:19.813230 27895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:19.814512 27895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:19.815761 27895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:19.828873 27903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:19.828997 27901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:19.828956 27900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:19.830608 27895 server_base.cc:1034] running on GCE node
I20260808 15:42:19.831144 27895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:19.832027 27895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:19.833223 27895 hybrid_clock.cc:648] HybridClock initialized: now 1786203739833178 us; error 43 us; skew 500 ppm
I20260808 15:42:19.833581 27895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:19.836988 27895 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:19.838487 27895 webserver.cc:460] Webserver started at http://127.23.130.3:37089/ using document root <none> and password file <none>
I20260808 15:42:19.839169 27895 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:19.839318 27895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:19.839630 27895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:19.842630 27895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "c7d5a613b72d436aab7d8afb6a8a5c5d"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "0d6b566789c32357838782c80fef241b"
server_key_iv: "a8033d670c736627b96515196d1786f7"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.843436 27895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "c7d5a613b72d436aab7d8afb6a8a5c5d"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "0d6b566789c32357838782c80fef241b"
server_key_iv: "a8033d670c736627b96515196d1786f7"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.848760 27895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:42:19.852252 27912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.853657 27895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:19.853790 27895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "c7d5a613b72d436aab7d8afb6a8a5c5d"
format_stamp: "Formatted at 2026-08-08 15:42:19 on dist-test-slave-gp6n"
server_key: "0d6b566789c32357838782c80fef241b"
server_key_iv: "a8033d670c736627b96515196d1786f7"
server_key_version: "encryptionkey@0"
I20260808 15:42:19.853998 27895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:19.868500 27895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:19.872248 27895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:19.872490 27895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:19.873345 27895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:19.874681 27895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:19.874754 27895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.874855 27895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:19.874899 27895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:19.908514 27895 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:43073
I20260808 15:42:19.908555 28034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:43073 every 8 connection(s)
I20260808 15:42:19.909884 27895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:19.912631 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 27895
I20260808 15:42:19.912854 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:19.913486 24072 external_mini_cluster.cc:1428] Setting key 27417c4da3e9097da9ada8e225c50e31
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:19.929143 27744 heartbeater.cc:503] Master 127.23.130.62:44183 was elected leader, sending a full tablet report...
I20260808 15:42:19.931674 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.912823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:57473 (local address 127.23.130.62:44183)
0808 15:42:19.913034 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:19.913039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:19.914679 (+ 1640us) server_negotiation.cc:407] Connection header received
0808 15:42:19.917117 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:19.917122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:19.917212 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:19.917319 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:19.918919 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.919722 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.921286 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.921710 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.921850 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:19.925361 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:19.925366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:19.925369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:19.925401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:19.928173 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.929098 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.929103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.929107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.929344 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.929923 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.929929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.929934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.930295 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:19.930500 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:19.931246 (+ 746us) server_negotiation.cc:299] Negotiation successful
0808 15:42:19.931427 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:42:19.932631 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.913509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44183 (local address 127.23.130.3:57473)
0808 15:42:19.914522 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:19.914565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:19.916804 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:19.917571 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:19.917579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:19.918088 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:19.918755 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:19.918774 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.919886 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.919896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:19.921131 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:19.921143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.921951 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.921959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:19.922202 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:19.922911 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:19.922933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:19.925167 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:19.928307 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:19.928314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:19.928326 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:19.928921 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:19.929489 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:19.929494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:19.929498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:19.929763 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:19.930477 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:19.930485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:19.930886 (+ 401us) client_negotiation.cc:769] Sending connection context
0808 15:42:19.931115 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 15:42:19.931501 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":372,"threads_started":1}
I20260808 15:42:19.934038 28035 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44183
I20260808 15:42:19.934381 28035 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:19.935195 28035 heartbeater.cc:511] Master 127.23.130.62:44183 requested a full tablet report, sending...
I20260808 15:42:19.936558 27541 ts_manager.cc:194] Registered new tserver with Master: c7d5a613b72d436aab7d8afb6a8a5c5d (127.23.130.3:43073)
I20260808 15:42:19.937285 27541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:57473
I20260808 15:42:19.941638 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 15:42:19 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:19.995309 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.981789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.23.130.62:44183)
0808 15:42:19.981971 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:19.981976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:19.982471 (+ 495us) server_negotiation.cc:407] Connection header received
0808 15:42:19.982679 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:19.982684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:19.982750 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:19.982844 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:19.984261 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.985001 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.986227 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:19.986676 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:19.986767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:19.990558 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:19.990563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:19.990566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:19.990599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:19.993128 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.993694 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.993697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.993700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.993823 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:19.994138 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:19.994141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:19.994143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:19.994502 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:19.994689 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:19.994914 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 15:42:19.995080 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 15:42:19.995898 27541 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35120
I20260808 15:42:19.998075 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27607
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:20.013895 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:19.999076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.23.130.1:44671)
0808 15:42:19.999716 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:19.999723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:19.999746 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:42:19.999799 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:19.999806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:20.000006 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:20.000189 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:20.001316 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.002314 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.003544 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.004114 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.004253 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:20.008178 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:20.008189 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:20.008204 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:20.008240 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:20.011055 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.011725 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.011733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.011735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.011867 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.012259 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.012264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.012268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.012637 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:20.012950 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:20.013251 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 15:42:20.013546 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
W20260808 15:42:20.015362 27738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47356
W20260808 15:42:20.016517 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:20.016712 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27607
I20260808 15:42:20.036020 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27750
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:20.052302 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:20.037004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.23.130.2:41371)
0808 15:42:20.037588 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:20.037596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:20.037621 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:42:20.037702 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:20.037708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:20.037914 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:20.038053 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:20.039381 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.040420 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.041712 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.042475 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.042587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:20.046897 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:20.046908 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:20.046922 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:20.046957 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:20.049671 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.050276 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.050282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.050285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.050530 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.050864 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.050867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.050869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.051216 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:20.051440 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:20.051701 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 15:42:20.051892 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":303,"threads_started":1}
W20260808 15:42:20.053682 27884 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33628
W20260808 15:42:20.055157 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27750: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:20.055315 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27750
I20260808 15:42:20.073967 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27895
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:20.090320 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:20.074990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.23.130.3:43073)
0808 15:42:20.075723 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:20.075730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:20.075759 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:42:20.075818 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:20.075824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:20.076032 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:20.076165 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:20.077536 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.078654 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.079892 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:20.080429 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:20.080528 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:20.084948 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:20.084959 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:20.084975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:20.085007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:20.087890 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.088551 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.088558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.088560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.088681 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:20.088972 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:20.088976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:20.088978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:20.089320 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:20.089600 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:20.089755 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 15:42:20.090031 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":404,"threads_started":1}
W20260808 15:42:20.091711 28028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37016
W20260808 15:42:20.092818 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:42:20.092981 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27895
I20260808 15:42:20.111596 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 27505
I20260808 15:42:20.112847 27572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:20.217626 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 27505
2026-08-08T15:42:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3727 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[28054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[28054](info): set up 2 sockets
Aug 08 15:42:20 dist-test-slave-gp6n krb5kdc[28054](info): commencing operation
krb5kdc: starting...
W20260808 15:42:21.377233 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:21 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:21.598814 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42259
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:21.928220 28068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:21.928500 28068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:21.928663 28068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:21.935887 28068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:21.936002 28068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:21.936105 28068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:21.936206 28068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:21.936270 28068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:21.945802 28068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:42259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:21.947141 28068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:21.948427 28068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:21.960433 28078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:21.960577 28076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:21.960548 28075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:21.961606 28068 server_base.cc:1034] running on GCE node
I20260808 15:42:21.962468 28068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:21.963878 28068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:21.965080 28068 hybrid_clock.cc:648] HybridClock initialized: now 1786203741965041 us; error 27 us; skew 500 ppm
I20260808 15:42:21.965438 28068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:21 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203741, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:21.972054 28068 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:21.973506 28068 webserver.cc:460] Webserver started at http://127.23.130.62:46847/ using document root <none> and password file <none>
I20260808 15:42:21.974195 28068 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:21.974354 28068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:21.974704 28068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:21.977663 28068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "48f5379313e94038a3aeadf8f0862ff3"
format_stamp: "Formatted at 2026-08-08 15:42:21 on dist-test-slave-gp6n"
server_key: "016fdb1ff8d662706d2d8d464a708d8b"
server_key_iv: "3e4e33e6b23027e69b431b3f60c5c8a2"
server_key_version: "encryptionkey@0"
I20260808 15:42:21.978554 28068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "48f5379313e94038a3aeadf8f0862ff3"
format_stamp: "Formatted at 2026-08-08 15:42:21 on dist-test-slave-gp6n"
server_key: "016fdb1ff8d662706d2d8d464a708d8b"
server_key_iv: "3e4e33e6b23027e69b431b3f60c5c8a2"
server_key_version: "encryptionkey@0"
I20260808 15:42:21.983613 28068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:21.987087 28086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:21.988451 28068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:21.988648 28068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "48f5379313e94038a3aeadf8f0862ff3"
format_stamp: "Formatted at 2026-08-08 15:42:21 on dist-test-slave-gp6n"
server_key: "016fdb1ff8d662706d2d8d464a708d8b"
server_key_iv: "3e4e33e6b23027e69b431b3f60c5c8a2"
server_key_version: "encryptionkey@0"
I20260808 15:42:21.988870 28068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:22.000792 28068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:22.004570 28068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:22.004858 28068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:22.023787 28068 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:42259
I20260808 15:42:22.023819 28142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:42259 every 8 connection(s)
I20260808 15:42:22.025274 28068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:22.029399 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:22.029935 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28068
I20260808 15:42:22.030165 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:22.031055 24072 external_mini_cluster.cc:1428] Setting key 2b45f135d2fc485a4707a76c605aa7a1
I20260808 15:42:22.043138 28143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3: Bootstrap starting.
I20260808 15:42:22.046401 28143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3: Neither blocks nor log segments found. Creating new log.
Aug 08 15:42:22 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:22.047765 28143 log.cc:824] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:22.051177 28143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3: No bootstrap required, opened a new log
I20260808 15:42:22.056299 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:22.033406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.23.130.62:42259)
0808 15:42:22.034293 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:22.034338 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:22.034389 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:42:22.036599 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:22.036639 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:22.037116 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:22.037625 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:22.039487 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.041005 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.042509 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.043298 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.043538 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:22.048217 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:22.048226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:22.048240 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:22.048308 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:22.051881 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:22.052767 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:22.052773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:22.052779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:22.052965 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:22.053383 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:22.053386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:22.053389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:22.054003 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:22.054370 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:22.054801 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 15:42:22.055185 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":397,"threads_started":1}
I20260808 15:42:22.059832 28143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } }
I20260808 15:42:22.060192 28143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:22.060298 28143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f5379313e94038a3aeadf8f0862ff3, State: Initialized, Role: FOLLOWER
I20260808 15:42:22.060877 28143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } }
I20260808 15:42:22.061161 28143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:22.061309 28143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:22.061482 28143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:22.063694 28143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } }
I20260808 15:42:22.064195 28143 leader_election.cc:302] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f5379313e94038a3aeadf8f0862ff3; no voters:
I20260808 15:42:22.064895 28143 leader_election.cc:288] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:22.065007 28149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:22.065730 28149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [term 1 LEADER]: Becoming Leader. State: Replica: 48f5379313e94038a3aeadf8f0862ff3, State: Running, Role: LEADER
I20260808 15:42:22.066344 28149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } }
I20260808 15:42:22.067260 28143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:22.069247 28151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f5379313e94038a3aeadf8f0862ff3. Latest consensus state: current_term: 1 leader_uuid: "48f5379313e94038a3aeadf8f0862ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } } }
I20260808 15:42:22.069700 28151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:22.070254 28150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f5379313e94038a3aeadf8f0862ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f5379313e94038a3aeadf8f0862ff3" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42259 } } }
I20260808 15:42:22.070694 28150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:22.072412 28159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:22.077502 28159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:22.086216 28159 catalog_manager.cc:1347] Generated new cluster ID: cc9b0d9339c0435bbdb40091c8f2ba82
I20260808 15:42:22.086362 28159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:22.100699 28159 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:22.101976 28159 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:22.113833 28159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f5379313e94038a3aeadf8f0862ff3: Generated new TSK 0
I20260808 15:42:22.114763 28159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:22.174391 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:22.510960 28172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:22.511270 28172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:22.511459 28172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:22.518544 28172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:22.518687 28172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:22.518872 28172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:22.528240 28172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:22.529524 28172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:22.530830 28172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:22.543941 28181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:22.544102 28178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:22.543824 28179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:22.545131 28172 server_base.cc:1034] running on GCE node
I20260808 15:42:22.545754 28172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:22.546782 28172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:22.548013 28172 hybrid_clock.cc:648] HybridClock initialized: now 1786203742547972 us; error 54 us; skew 500 ppm
I20260808 15:42:22.548336 28172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:22 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:22.551831 28172 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:22.553282 28172 webserver.cc:460] Webserver started at http://127.23.130.1:39191/ using document root <none> and password file <none>
I20260808 15:42:22.553995 28172 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:22.554148 28172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:22.554550 28172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:22.557783 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "b6092935b4f24accb96dc0f05114c86e"
format_stamp: "Formatted at 2026-08-08 15:42:22 on dist-test-slave-gp6n"
server_key: "0d91d3c537028ab726ea2410f166793d"
server_key_iv: "1bc71d20643f1bd317fb4562c967e211"
server_key_version: "encryptionkey@0"
I20260808 15:42:22.558660 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "b6092935b4f24accb96dc0f05114c86e"
format_stamp: "Formatted at 2026-08-08 15:42:22 on dist-test-slave-gp6n"
server_key: "0d91d3c537028ab726ea2410f166793d"
server_key_iv: "1bc71d20643f1bd317fb4562c967e211"
server_key_version: "encryptionkey@0"
I20260808 15:42:22.563875 28172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 15:42:22.567422 28188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:22.568840 28172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:42:22.569017 28172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "b6092935b4f24accb96dc0f05114c86e"
format_stamp: "Formatted at 2026-08-08 15:42:22 on dist-test-slave-gp6n"
server_key: "0d91d3c537028ab726ea2410f166793d"
server_key_iv: "1bc71d20643f1bd317fb4562c967e211"
server_key_version: "encryptionkey@0"
I20260808 15:42:22.569232 28172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:22.587390 28172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:22.591858 28172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:22.592177 28172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:22.593230 28172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:22.594731 28172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:22.594836 28172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:22.595005 28172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:22.595080 28172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:22.629632 28172 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:35827
I20260808 15:42:22.629694 28312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:35827 every 8 connection(s)
I20260808 15:42:22.631057 28172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:22.638154 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28172
I20260808 15:42:22.638366 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:22.639134 24072 external_mini_cluster.cc:1428] Setting key 27bbf9ef1d28a09d0cc00e3adb4c5317
Aug 08 15:42:22 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:22.650945 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:22.633653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:34643 (local address 127.23.130.62:42259)
0808 15:42:22.634098 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:22.634102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:22.635179 (+ 1077us) server_negotiation.cc:407] Connection header received
0808 15:42:22.637017 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:22.637023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:22.637122 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:22.637246 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:22.638899 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.639685 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.641211 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.641613 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.641752 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:22.645193 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:22.645199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:22.645202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:22.645233 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:22.647848 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:22.648587 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:22.648599 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:22.648603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:22.648749 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:22.649205 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:22.649209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:22.649211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:22.649475 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:22.649634 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:22.650328 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 15:42:22.650588 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260808 15:42:22.651976 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:22.634226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42259 (local address 127.23.130.1:34643)
0808 15:42:22.635008 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:22.635051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:22.636746 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:22.637490 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:22.637499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:22.638005 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:22.638722 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:22.638743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.639838 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.639844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:22.641064 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:22.641076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:22.641845 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:22.641855 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:22.642089 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:22.642813 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:22.642840 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:22.644960 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:22.647998 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:22.648008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:22.648024 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:22.648445 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:22.648868 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:22.648873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:22.648877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:22.649074 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:22.649606 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:22.649614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:22.649949 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 15:42:22.650215 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:42:22.650780 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260808 15:42:22.653555 28313 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42259
I20260808 15:42:22.653877 28313 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:22.654661 28313 heartbeater.cc:511] Master 127.23.130.62:42259 requested a full tablet report, sending...
I20260808 15:42:22.656702 28105 ts_manager.cc:194] Registered new tserver with Master: b6092935b4f24accb96dc0f05114c86e (127.23.130.1:35827)
I20260808 15:42:22.658164 28105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:34643
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:22.689503 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:23.018705 28320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:23.019004 28320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:23.019169 28320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:23.026572 28320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:23.026690 28320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:23.026870 28320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:23.036885 28320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:23.038218 28320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:23.039669 28320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:23.053143 28325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:23.053256 28328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:23.053216 28326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:23.054206 28320 server_base.cc:1034] running on GCE node
I20260808 15:42:23.054858 28320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:23.055789 28320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:23.056986 28320 hybrid_clock.cc:648] HybridClock initialized: now 1786203743056928 us; error 58 us; skew 500 ppm
I20260808 15:42:23.057296 28320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:23.060472 28320 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:23.061896 28320 webserver.cc:460] Webserver started at http://127.23.130.2:36939/ using document root <none> and password file <none>
I20260808 15:42:23.062678 28320 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:23.062786 28320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:23.063056 28320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:23.065933 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "36e9c99dc30046098317840df09f0e0f"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "c66564356677a9073c7601381ea4661a"
server_key_iv: "2c06c286fd43f47a8d18768c52662966"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.066732 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "36e9c99dc30046098317840df09f0e0f"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "c66564356677a9073c7601381ea4661a"
server_key_iv: "2c06c286fd43f47a8d18768c52662966"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.072280 28320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:23.075987 28337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.077262 28320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:23.077395 28320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "36e9c99dc30046098317840df09f0e0f"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "c66564356677a9073c7601381ea4661a"
server_key_iv: "2c06c286fd43f47a8d18768c52662966"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.077590 28320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:23.099884 28320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:23.104321 28320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:23.104585 28320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:23.105481 28320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:23.106865 28320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:23.106942 28320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.107030 28320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:23.107079 28320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.140269 28320 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:44181
I20260808 15:42:23.140386 28458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:44181 every 8 connection(s)
I20260808 15:42:23.141625 28320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:23.150957 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28320
I20260808 15:42:23.151147 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:23.151693 24072 external_mini_cluster.cc:1428] Setting key ec4f4e1f4c5d832d165c2b12348e4c30
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:23.163798 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.144283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:52373 (local address 127.23.130.62:42259)
0808 15:42:23.144500 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.144506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.145684 (+ 1178us) server_negotiation.cc:407] Connection header received
0808 15:42:23.147438 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.147442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.147525 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.147617 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.149048 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.149772 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.151480 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.152280 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.152445 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.156340 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.156349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.156357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.156402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.160204 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.160935 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.160943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.160956 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.161125 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.161634 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.161641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.161647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.162030 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.162194 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.163068 (+ 874us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.163362 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:42:23.164798 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.144811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42259 (local address 127.23.130.2:52373)
0808 15:42:23.145519 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:23.145560 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:23.147192 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:23.147813 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:23.147821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:23.148332 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:23.148895 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:23.148912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.149916 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.149921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:23.151340 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:23.151353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.152549 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.152559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:23.152798 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.153498 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:23.153521 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:23.156111 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:23.160354 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:23.160360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:23.160373 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:23.160804 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:23.161249 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:23.161255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:23.161261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:23.161507 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:23.162157 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:23.162167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:23.162647 (+ 480us) client_negotiation.cc:769] Sending connection context
0808 15:42:23.162908 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 15:42:23.163350 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":272,"threads_started":1}
I20260808 15:42:23.166800 28459 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42259
I20260808 15:42:23.167129 28459 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:23.167909 28459 heartbeater.cc:511] Master 127.23.130.62:42259 requested a full tablet report, sending...
I20260808 15:42:23.169507 28105 ts_manager.cc:194] Registered new tserver with Master: 36e9c99dc30046098317840df09f0e0f (127.23.130.2:44181)
I20260808 15:42:23.170805 28105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:52373
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:23.206169 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:23.543632 28465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:23.543890 28465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:23.543992 28465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:23.551132 28465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:23.551230 28465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:23.551339 28465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:23.560855 28465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:23.562052 28465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:23.563474 28465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:23.576903 28471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:23.577034 28470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:23.577059 28473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:23.578115 28465 server_base.cc:1034] running on GCE node
I20260808 15:42:23.578742 28465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:23.579607 28465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:23.580803 28465 hybrid_clock.cc:648] HybridClock initialized: now 1786203743580754 us; error 51 us; skew 500 ppm
I20260808 15:42:23.581125 28465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:23.584319 28465 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:23.585767 28465 webserver.cc:460] Webserver started at http://127.23.130.3:43463/ using document root <none> and password file <none>
I20260808 15:42:23.586385 28465 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:23.586560 28465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:23.586817 28465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:23.589634 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "51eaf926bec549bdabdfb093756f2d37"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "6f5b227ab46f96bec9e4a08cf7cef363"
server_key_iv: "c73514c9366a8be2e34d83adca398ed5"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.590377 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "51eaf926bec549bdabdfb093756f2d37"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "6f5b227ab46f96bec9e4a08cf7cef363"
server_key_iv: "c73514c9366a8be2e34d83adca398ed5"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.595650 28465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:42:23.599014 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.600344 28465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:23.600517 28465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "51eaf926bec549bdabdfb093756f2d37"
format_stamp: "Formatted at 2026-08-08 15:42:23 on dist-test-slave-gp6n"
server_key: "6f5b227ab46f96bec9e4a08cf7cef363"
server_key_iv: "c73514c9366a8be2e34d83adca398ed5"
server_key_version: "encryptionkey@0"
I20260808 15:42:23.600699 28465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:23.615125 28465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:23.619110 28465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:23.619393 28465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:23.620309 28465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:23.621603 28465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:23.621685 28465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.621806 28465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:23.621860 28465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:23.656096 28465 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:46111
I20260808 15:42:23.656186 28604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:46111 every 8 connection(s)
I20260808 15:42:23.657429 28465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:23.659256 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28465
I20260808 15:42:23.659502 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:23.660107 24072 external_mini_cluster.cc:1428] Setting key 457108509e45bc94e3ce8aa6dde4d949
I20260808 15:42:23.661055 28313 heartbeater.cc:503] Master 127.23.130.62:42259 was elected leader, sending a full tablet report...
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:23.678191 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.660363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:41221 (local address 127.23.130.62:42259)
0808 15:42:23.660545 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.660550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.661963 (+ 1413us) server_negotiation.cc:407] Connection header received
0808 15:42:23.664107 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.664112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.664192 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.664308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.665768 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.666632 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.668257 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.668693 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.668828 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.672279 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.672284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.672287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.672377 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.675256 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.675963 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.675969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.675973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.676110 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.676528 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.676535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.676538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.676803 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.676956 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.677621 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.677839 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:42:23.679188 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.660899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42259 (local address 127.23.130.3:41221)
0808 15:42:23.661782 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:23.661855 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:23.663849 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:23.664555 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:23.664563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:23.665056 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:23.665603 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:23.665620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.666778 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.666783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:23.668110 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:23.668122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.668921 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.668930 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:23.669173 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.669806 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:23.669827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:23.672113 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:23.675401 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:23.675410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:23.675432 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:23.675822 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:23.676222 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:23.676228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:23.676231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:23.676417 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:23.676939 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:23.676947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:23.677261 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 15:42:23.677527 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:42:23.678041 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":317,"threads_started":1}
I20260808 15:42:23.680531 28605 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42259
I20260808 15:42:23.680840 28605 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:23.681584 28605 heartbeater.cc:511] Master 127.23.130.62:42259 requested a full tablet report, sending...
I20260808 15:42:23.682999 28105 ts_manager.cc:194] Registered new tserver with Master: 51eaf926bec549bdabdfb093756f2d37 (127.23.130.3:46111)
I20260808 15:42:23.683737 28105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:41221
I20260808 15:42:23.687870 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:23.742623 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.727662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.23.130.1:35827)
0808 15:42:23.728317 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.728326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.728353 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:42:23.728415 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.728422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.728653 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.728785 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.730164 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.731260 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.732495 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.733044 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.733141 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.737122 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.737133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.737148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.737182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.740032 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.740695 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.740703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.740705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.740831 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.741174 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.741177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.741179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.741515 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.741746 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.742016 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.742249 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":330,"threads_started":1}
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:23.760414 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.747029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.23.130.62:42259)
0808 15:42:23.747268 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.747272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.747378 (+ 106us) server_negotiation.cc:407] Connection header received
0808 15:42:23.747471 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.747474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.747556 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.747649 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.748879 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.749647 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.750910 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.751340 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.751448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.755302 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.755308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.755311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.755346 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.757950 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.758711 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.758715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.758717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.758851 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.759174 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.759177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.759179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.759483 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.759663 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.759974 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.760186 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 15:42:23.761022 28105 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40910
I20260808 15:42:23.762094 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28172
I20260808 15:42:23.775197 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.763195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.23.130.1:35827)
0808 15:42:23.763411 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.763418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.763573 (+ 155us) server_negotiation.cc:407] Connection header received
0808 15:42:23.763765 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.763769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.763833 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.763953 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.765128 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.765793 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.767137 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.767657 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.767817 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.770059 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.770063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.770067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.770096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.772894 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.773540 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.773544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.773547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.773669 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.773979 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.773982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.773984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.774211 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.774383 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.774688 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.774845 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:42:23.775787 28307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:23.858790 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28172
I20260808 15:42:23.878080 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28320
Aug 08 15:42:23 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:23.894173 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:23.879100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.23.130.2:44181)
0808 15:42:23.879649 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:23.879656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:23.879683 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:42:23.879751 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:23.879757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:23.879978 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:23.880112 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:23.881424 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.882586 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.883790 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:23.884325 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:23.884420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:23.888622 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:23.888633 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:23.888646 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:23.888678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:23.891444 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.892120 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.892126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.892129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.892255 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:23.892613 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:23.892616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:23.892618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:23.892998 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:23.893227 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:23.893578 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 15:42:23.893814 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":291,"threads_started":1}
I20260808 15:42:23.895533 28454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:23.978608 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28320
I20260808 15:42:23.999155 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28465
Aug 08 15:42:24 dist-test-slave-gp6n krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:24.017429 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:24.000149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.23.130.3:46111)
0808 15:42:24.000843 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:24.000850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:24.000881 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:42:24.000951 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:24.000958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:24.001177 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:24.001373 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:24.002868 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:24.004110 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:24.005631 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:24.006254 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:24.006452 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:24.011223 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:24.011235 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:24.011249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:24.011283 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:24.014150 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:24.014940 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:24.014948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:24.014950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:24.015079 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:24.015511 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:24.015515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:24.015517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:24.016069 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:24.016423 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:24.016740 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 15:42:24.017041 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":359,"threads_started":1}
I20260808 15:42:24.018771 28599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:24.102065 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28465
I20260808 15:42:24.118156 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28068
I20260808 15:42:24.119498 28138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:24.228971 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28068
2026-08-08T15:42:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4011 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:24 dist-test-slave-gp6n krb5kdc[28626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:24 dist-test-slave-gp6n krb5kdc[28626](info): set up 2 sockets
Aug 08 15:42:24 dist-test-slave-gp6n krb5kdc[28626](info): commencing operation
krb5kdc: starting...
W20260808 15:42:25.397349 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:25 dist-test-slave-gp6n krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:25.622824 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42953
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:25.952623 28639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:25.952899 28639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:25.953078 28639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:25.960242 28639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:25.960382 28639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:25.960471 28639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:25.960580 28639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:25.960667 28639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:25.970288 28639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:25.971597 28639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:25.972920 28639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:25.985508 28645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:25.985577 28646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:25.985600 28648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:25.986647 28639 server_base.cc:1034] running on GCE node
I20260808 15:42:25.987434 28639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:25.988863 28639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:25.990115 28639 hybrid_clock.cc:648] HybridClock initialized: now 1786203745990070 us; error 54 us; skew 500 ppm
I20260808 15:42:25.990538 28639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:25 dist-test-slave-gp6n krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:25.993749 28639 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:25.994912 28639 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:25.995004 28639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:25.995283 28639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:25.998096 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "f16e032796704dad8cf515a54f49a235"
format_stamp: "Formatted at 2026-08-08 15:42:25 on dist-test-slave-gp6n"
server_key: "6f8fec4b789c77f6383c6e4dc00b6253"
server_key_iv: "988f7db473d283c56675fbbb5d8502d4"
server_key_version: "encryptionkey@0"
I20260808 15:42:25.998903 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "f16e032796704dad8cf515a54f49a235"
format_stamp: "Formatted at 2026-08-08 15:42:25 on dist-test-slave-gp6n"
server_key: "6f8fec4b789c77f6383c6e4dc00b6253"
server_key_iv: "988f7db473d283c56675fbbb5d8502d4"
server_key_version: "encryptionkey@0"
I20260808 15:42:26.003795 28639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:42:26.007081 28654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:26.008325 28639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:26.008481 28639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "f16e032796704dad8cf515a54f49a235"
format_stamp: "Formatted at 2026-08-08 15:42:25 on dist-test-slave-gp6n"
server_key: "6f8fec4b789c77f6383c6e4dc00b6253"
server_key_iv: "988f7db473d283c56675fbbb5d8502d4"
server_key_version: "encryptionkey@0"
I20260808 15:42:26.008659 28639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:26.038220 28639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:26.043098 28639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:26.043340 28639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:26.062570 28639 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:42953
I20260808 15:42:26.062600 28710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:42953 every 8 connection(s)
I20260808 15:42:26.063551 28639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:26.064311 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28639
I20260808 15:42:26.064543 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:26.065333 24072 external_mini_cluster.cc:1428] Setting key 45a5c66152b65ddc12164467ea214879
I20260808 15:42:26.068075 28711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:26.078984 28711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: Bootstrap starting.
Aug 08 15:42:26 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:26.082883 28711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:26.083981 28711 log.cc:824] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:26.086861 28711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: No bootstrap required, opened a new log
I20260808 15:42:26.088235 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:26.067502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.23.130.62:42953)
0808 15:42:26.068399 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:26.068412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:26.068472 (+ 60us) server_negotiation.cc:407] Connection header received
0808 15:42:26.070112 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:26.070133 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:26.070607 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:26.071017 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:26.072608 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.074014 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.075360 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.076088 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.076337 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:26.080866 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:26.080874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:26.080887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:26.080925 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:26.084024 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:26.084749 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:26.084755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:26.084761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:26.084930 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:26.085374 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:26.085381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:26.085389 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:26.085910 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:26.086234 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:26.086705 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 15:42:26.087277 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":401,"threads_started":1}
I20260808 15:42:26.094906 28711 raft_consensus.cc:348] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } }
I20260808 15:42:26.095288 28711 raft_consensus.cc:374] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:26.095402 28711 raft_consensus.cc:729] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16e032796704dad8cf515a54f49a235, State: Initialized, Role: FOLLOWER
I20260808 15:42:26.095952 28711 consensus_queue.cc:260] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } }
I20260808 15:42:26.096220 28711 raft_consensus.cc:388] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:26.096382 28711 raft_consensus.cc:482] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:26.096578 28711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:26.098798 28711 raft_consensus.cc:504] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } }
I20260808 15:42:26.099220 28711 leader_election.cc:302] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f16e032796704dad8cf515a54f49a235; no voters:
I20260808 15:42:26.099769 28711 leader_election.cc:288] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:26.099937 28717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:26.100746 28717 raft_consensus.cc:686] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [term 1 LEADER]: Becoming Leader. State: Replica: f16e032796704dad8cf515a54f49a235, State: Running, Role: LEADER
I20260808 15:42:26.101308 28717 consensus_queue.cc:237] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } }
I20260808 15:42:26.102172 28711 sys_catalog.cc:564] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:26.104331 28718 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f16e032796704dad8cf515a54f49a235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } } }
I20260808 15:42:26.104765 28718 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:26.105381 28719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f16e032796704dad8cf515a54f49a235. Latest consensus state: current_term: 1 leader_uuid: "f16e032796704dad8cf515a54f49a235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e032796704dad8cf515a54f49a235" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42953 } } }
I20260808 15:42:26.105697 28719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:26.106262 28728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:26.111217 28728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:26.121460 28728 catalog_manager.cc:1347] Generated new cluster ID: 8e8261188bc84c3c853f32ba294c5530
I20260808 15:42:26.121574 28728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:26.168516 28728 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:26.170176 28728 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:26.179551 28728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: Generated new TSK 0
I20260808 15:42:26.180394 28728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:26.250903 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:26.604094 28740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:26.604362 28740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:26.604533 28740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:26.612191 28740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:26.612317 28740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:26.612496 28740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:26.622193 28740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:26.623522 28740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:26.624835 28740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:26.638213 28745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:26.638152 28749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:26.638183 28746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:26.639290 28740 server_base.cc:1034] running on GCE node
I20260808 15:42:26.639838 28740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:26.640774 28740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:26.641971 28740 hybrid_clock.cc:648] HybridClock initialized: now 1786203746641920 us; error 51 us; skew 500 ppm
I20260808 15:42:26.642241 28740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:26 dist-test-slave-gp6n krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:26.645546 28740 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:26.646654 28740 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:26.646747 28740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:26.647018 28740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:26.649904 28740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "05a9b93970454abe91391cb7797fd3ef"
format_stamp: "Formatted at 2026-08-08 15:42:26 on dist-test-slave-gp6n"
server_key: "5d03e088314178b4c390bf6b543eceb7"
server_key_iv: "1f4498c163b6f42e5fd2e00da326fb17"
server_key_version: "encryptionkey@0"
I20260808 15:42:26.650727 28740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "05a9b93970454abe91391cb7797fd3ef"
format_stamp: "Formatted at 2026-08-08 15:42:26 on dist-test-slave-gp6n"
server_key: "5d03e088314178b4c390bf6b543eceb7"
server_key_iv: "1f4498c163b6f42e5fd2e00da326fb17"
server_key_version: "encryptionkey@0"
I20260808 15:42:26.655576 28740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:26.658880 28755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:26.660221 28740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:42:26.660370 28740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "05a9b93970454abe91391cb7797fd3ef"
format_stamp: "Formatted at 2026-08-08 15:42:26 on dist-test-slave-gp6n"
server_key: "5d03e088314178b4c390bf6b543eceb7"
server_key_iv: "1f4498c163b6f42e5fd2e00da326fb17"
server_key_version: "encryptionkey@0"
I20260808 15:42:26.660620 28740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:26.675895 28740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:26.680251 28740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:26.680533 28740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:26.681455 28740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:26.682924 28740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:26.683012 28740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:26.683091 28740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:26.683151 28740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:26.716614 28740 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:39317
I20260808 15:42:26.716657 28880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:39317 every 8 connection(s)
I20260808 15:42:26.717466 28740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:26.726873 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28740
I20260808 15:42:26.727080 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:26.727901 24072 external_mini_cluster.cc:1428] Setting key 7729caa21b6b529ee9ba95417e14e49d
Aug 08 15:42:26 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:26.738226 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:26.719893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:45785 (local address 127.23.130.62:42953)
0808 15:42:26.720366 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:26.720370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:26.721187 (+ 817us) server_negotiation.cc:407] Connection header received
0808 15:42:26.723085 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:26.723091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:26.723185 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:26.723308 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:26.724853 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.725572 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.727713 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.728170 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.728275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:26.732476 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:26.732481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:26.732485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:26.732518 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:26.735179 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:26.735921 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:26.735929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:26.735937 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:26.736053 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:26.736481 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:26.736486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:26.736489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:26.736754 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:26.736916 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:26.737627 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 15:42:26.737850 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260808 15:42:26.739245 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:26.720298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42953 (local address 127.23.130.1:45785)
0808 15:42:26.721019 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:26.721058 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:26.722804 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:26.723543 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:26.723551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:26.724088 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:26.724657 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:26.724678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.725734 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.725739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:26.727548 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:26.727563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:26.728409 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:26.728417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:26.728655 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:26.729555 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:26.729578 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:26.732246 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:26.735326 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:26.735335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:26.735351 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:26.735787 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:26.736167 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:26.736171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:26.736175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:26.736358 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:26.736897 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:26.736905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:26.737245 (+ 340us) client_negotiation.cc:769] Sending connection context
0808 15:42:26.737511 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:42:26.738060 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
I20260808 15:42:26.740891 28881 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42953
I20260808 15:42:26.741266 28881 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:26.742101 28881 heartbeater.cc:511] Master 127.23.130.62:42953 requested a full tablet report, sending...
I20260808 15:42:26.744585 28671 ts_manager.cc:194] Registered new tserver with Master: 05a9b93970454abe91391cb7797fd3ef (127.23.130.1:39317)
I20260808 15:42:26.746899 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:45785
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:26.782097 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:27.121411 28887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:27.121775 28887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:27.121971 28887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:27.130116 28887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:27.130296 28887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:27.130551 28887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:27.141330 28887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:27.142824 28887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:27.144263 28887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:27.157572 28893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:27.157625 28896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:27.157646 28894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:27.158267 28887 server_base.cc:1034] running on GCE node
I20260808 15:42:27.158928 28887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:27.159886 28887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:27.161127 28887 hybrid_clock.cc:648] HybridClock initialized: now 1786203747161071 us; error 45 us; skew 500 ppm
I20260808 15:42:27.161510 28887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:27 dist-test-slave-gp6n krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:27.168828 28887 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:27.170012 28887 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:27.170146 28887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:27.170534 28887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:27.173461 28887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "dad4a5fdfe8048568b89dae450b48d8e"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "132be1f8dbd57ed3ad8c0cacae2ac62f"
server_key_iv: "21b5f4fd928fbe282fbba2df047e06c6"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.174284 28887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "dad4a5fdfe8048568b89dae450b48d8e"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "132be1f8dbd57ed3ad8c0cacae2ac62f"
server_key_iv: "21b5f4fd928fbe282fbba2df047e06c6"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.179924 28887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:27.184019 28902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.185429 28887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:27.185658 28887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "dad4a5fdfe8048568b89dae450b48d8e"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "132be1f8dbd57ed3ad8c0cacae2ac62f"
server_key_iv: "21b5f4fd928fbe282fbba2df047e06c6"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.185916 28887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:27.197409 28887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:27.202054 28887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:27.202322 28887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:27.203431 28887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:27.204804 28887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:27.204931 28887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.205065 28887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:27.205161 28887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.239657 28887 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:45577
I20260808 15:42:27.239748 29027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:45577 every 8 connection(s)
I20260808 15:42:27.240736 28887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:27.243753 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 28887
I20260808 15:42:27.243955 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:27.244643 24072 external_mini_cluster.cc:1428] Setting key 3901cbd2f1ff54f987a626868400ec05
Aug 08 15:42:27 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:27.263267 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.244077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:38993 (local address 127.23.130.62:42953)
0808 15:42:27.244632 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.244640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.245589 (+ 949us) server_negotiation.cc:407] Connection header received
0808 15:42:27.247620 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.247626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.247720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.247851 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:27.249492 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.250282 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.252000 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.252480 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.252632 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.256735 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:27.256740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:27.256744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:27.256777 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:27.259691 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.260637 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.260641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.260643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.260778 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.261317 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.261320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.261323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.261746 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:27.261868 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.262827 (+ 959us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.262994 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
I20260808 15:42:27.264855 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.244067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42953 (local address 127.23.130.2:38993)
0808 15:42:27.245053 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:27.245105 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:27.247337 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:27.248095 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:27.248103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:27.248682 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:27.249302 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.249322 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.250481 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.250486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.251841 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.251854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.252725 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.252732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.252973 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.253736 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:27.253758 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:27.256505 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:27.259841 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:27.259852 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:27.259870 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:27.260510 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:27.260896 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:27.260904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:27.260911 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:27.261197 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:27.261879 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:27.261889 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:27.262350 (+ 461us) client_negotiation.cc:769] Sending connection context
0808 15:42:27.262719 (+ 369us) client_negotiation.cc:240] Negotiation successful
0808 15:42:27.263296 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":441,"threads_started":1}
I20260808 15:42:27.266475 29029 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42953
I20260808 15:42:27.266800 29029 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:27.267515 29029 heartbeater.cc:511] Master 127.23.130.62:42953 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:42:27.269088 28671 ts_manager.cc:194] Registered new tserver with Master: dad4a5fdfe8048568b89dae450b48d8e (127.23.130.2:45577)
I20260808 15:42:27.270016 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:38993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:27.297155 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:27.630815 29035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:27.631093 29035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:27.631261 29035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:27.638257 29035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:27.638360 29035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:27.638568 29035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:27.647976 29035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:27.649230 29035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:27.650574 29035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:27.664433 29041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:27.664527 29043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:27.664991 29040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:27.665136 29035 server_base.cc:1034] running on GCE node
I20260808 15:42:27.665793 29035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:27.666751 29035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:27.667966 29035 hybrid_clock.cc:648] HybridClock initialized: now 1786203747667927 us; error 51 us; skew 500 ppm
I20260808 15:42:27.668274 29035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:27 dist-test-slave-gp6n krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:27.671331 29035 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:27.672442 29035 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:27.672575 29035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:27.672891 29035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:27.675971 29035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "35768e019b0449e0abcbface2d505384"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "522a94234add5c417685dd71f5774f22"
server_key_iv: "029c0e289b9595c1444eb7a82932c5ee"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.676781 29035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "35768e019b0449e0abcbface2d505384"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "522a94234add5c417685dd71f5774f22"
server_key_iv: "029c0e289b9595c1444eb7a82932c5ee"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.681722 29035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:42:27.685048 29052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.686288 29035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:27.686501 29035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "35768e019b0449e0abcbface2d505384"
format_stamp: "Formatted at 2026-08-08 15:42:27 on dist-test-slave-gp6n"
server_key: "522a94234add5c417685dd71f5774f22"
server_key_iv: "029c0e289b9595c1444eb7a82932c5ee"
server_key_version: "encryptionkey@0"
I20260808 15:42:27.686718 29035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:27.734679 29035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:27.738766 29035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:27.739044 29035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:27.740157 29035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:27.742288 29035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:27.742437 29035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.742595 29035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:27.742671 29035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:27.750927 28881 heartbeater.cc:503] Master 127.23.130.62:42953 was elected leader, sending a full tablet report...
I20260808 15:42:27.778031 29035 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:35499
I20260808 15:42:27.778122 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:35499 every 8 connection(s)
I20260808 15:42:27.778961 29035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:27.779783 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29035
I20260808 15:42:27.779963 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:27.780514 24072 external_mini_cluster.cc:1428] Setting key 7800be0960f7766b5caff75bdf5d6508
Aug 08 15:42:27 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:27.803239 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.781777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:33317 (local address 127.23.130.62:42953)
0808 15:42:27.788052 (+ 6275us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.788056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.788073 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:27.788920 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.788924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.789003 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.789095 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:27.790803 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.791718 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.793213 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.793622 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.793715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.797344 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:27.797349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:27.797352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:27.797596 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:27.800342 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.801006 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.801010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.801012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.801128 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.801551 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.801555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.801557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.801827 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:27.801960 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.802712 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.802904 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6176,"thread_start_us":6042,"threads_started":1}
I20260808 15:42:27.804220 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.782554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42953 (local address 127.23.130.3:33317)
0808 15:42:27.786958 (+ 4404us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:27.786999 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:27.788688 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:27.789326 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:27.789334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:27.789986 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:27.790638 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.790658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.791854 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.791857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.793097 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.793108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.793819 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.793823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.794029 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.794832 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:27.794855 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:27.797154 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:27.800467 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:27.800473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:27.800485 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:27.800897 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:27.801237 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:27.801240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:27.801243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:27.801410 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:27.801967 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:27.801972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:27.802283 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 15:42:27.802560 (+ 277us) client_negotiation.cc:240] Negotiation successful
0808 15:42:27.802999 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4154,"thread_start_us":3921,"threads_started":1}
I20260808 15:42:27.805730 29176 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42953
I20260808 15:42:27.806046 29176 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:27.806929 29176 heartbeater.cc:511] Master 127.23.130.62:42953 requested a full tablet report, sending...
I20260808 15:42:27.808501 28671 ts_manager.cc:194] Registered new tserver with Master: 35768e019b0449e0abcbface2d505384 (127.23.130.3:35499)
I20260808 15:42:27.809376 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:33317
I20260808 15:42:27.820303 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:42:27.838860 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.825247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.23.130.62:42953)
0808 15:42:27.825584 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.825593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.825999 (+ 406us) server_negotiation.cc:407] Connection header received
0808 15:42:27.826251 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.826255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.826326 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.826478 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:27.828101 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.828911 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.830356 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.830872 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.831018 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.833574 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:27.833579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:27.833583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:27.833616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:27.836187 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.836866 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.836873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.836882 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.837001 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:27.837376 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:27.837382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:27.837386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:27.837711 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:27.837952 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.838310 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.838581 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:42:27.842900 28671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:27.849756 28671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:27.872677 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.864047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.23.130.1:39317)
0808 15:42:27.865247 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.865257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.865289 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:42:27.865397 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.865405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.865623 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.866512 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:27.866698 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.867867 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.870552 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871496 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.871660 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.871906 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.872164 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.872337 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":806,"threads_started":1}
I20260808 15:42:27.872766 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.863711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:39317 (local address 127.0.0.1:46600)
0808 15:42:27.864363 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:27.864386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:27.864526 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:27.865830 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:27.865835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:27.865860 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:27.866272 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.866294 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.868018 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.868023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.870380 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.870390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871699 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.871705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.871903 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.871966 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 15:42:27.872125 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 15:42:27.872317 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":356,"threads_started":1}
I20260808 15:42:27.873291 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.862040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:35499 (local address 127.0.0.1:49200)
0808 15:42:27.863215 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:27.863234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:27.863346 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:27.864987 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:27.864996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:27.865022 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:27.865429 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.865441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.867604 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.867611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.870156 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.870171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871991 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.871997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.872214 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.872276 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 15:42:27.872359 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:42:27.872493 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":96,"thread_start_us":882,"threads_started":1}
I20260808 15:42:27.873907 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.862760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.23.130.2:45577)
0808 15:42:27.863898 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.863909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.863947 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:42:27.864080 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.864088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.864340 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.865393 (+ 1053us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:27.865635 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.867301 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.870209 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871716 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.872295 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.872520 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.873397 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.873586 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":902,"threads_started":1}
I20260808 15:42:27.874066 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.861948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:45577 (local address 127.0.0.1:47312)
0808 15:42:27.862852 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:27.862901 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:27.863104 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:27.864585 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:27.864605 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:27.864644 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:27.865106 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.865125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.867447 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.867452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.870045 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:27.870057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871979 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.871985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:27.872213 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.872603 (+ 390us) client_negotiation.cc:769] Sending connection context
0808 15:42:27.872706 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:42:27.872826 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":441,"thread_start_us":1066,"threads_started":1}
I20260808 15:42:27.874855 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:27.862765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.23.130.3:35499)
0808 15:42:27.864352 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:27.864361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:27.864397 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:42:27.864504 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:27.864513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:27.864747 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:27.865636 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:27.865934 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.867471 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.870308 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:27.871716 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:27.872985 (+ 1269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:27.873210 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:27.873921 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 15:42:27.874534 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":723,"threads_started":1}
I20260808 15:42:27.877352 28811 tablet_service.cc:1467] Processing CreateTablet for tablet 9275cfdc572b429382fce788a4ee4573 (DEFAULT_TABLE table=test-table [id=68b5654f95164b6f92490435e6cdac28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:27.877579 29107 tablet_service.cc:1467] Processing CreateTablet for tablet 9275cfdc572b429382fce788a4ee4573 (DEFAULT_TABLE table=test-table [id=68b5654f95164b6f92490435e6cdac28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:27.877640 28958 tablet_service.cc:1467] Processing CreateTablet for tablet 9275cfdc572b429382fce788a4ee4573 (DEFAULT_TABLE table=test-table [id=68b5654f95164b6f92490435e6cdac28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:27.878980 28811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9275cfdc572b429382fce788a4ee4573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:27.878989 29107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9275cfdc572b429382fce788a4ee4573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:27.878997 28958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9275cfdc572b429382fce788a4ee4573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:27.890187 29197 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Bootstrap starting.
I20260808 15:42:27.893587 29199 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Bootstrap starting.
I20260808 15:42:27.895139 29197 tablet_bootstrap.cc:654] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:27.896499 29197 log.cc:824] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:27.897260 29199 tablet_bootstrap.cc:654] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:27.898002 29198 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Bootstrap starting.
I20260808 15:42:27.898628 29199 log.cc:824] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:27.900247 29197 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: No bootstrap required, opened a new log
I20260808 15:42:27.900574 29197 ts_tablet_manager.cc:1397] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:42:27.901443 29198 tablet_bootstrap.cc:654] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:27.901678 29199 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: No bootstrap required, opened a new log
I20260808 15:42:27.901979 29199 ts_tablet_manager.cc:1397] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 15:42:27.902635 29198 log.cc:824] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:27.905303 29198 tablet_bootstrap.cc:492] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: No bootstrap required, opened a new log
I20260808 15:42:27.905576 29198 ts_tablet_manager.cc:1397] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 15:42:27.911450 29197 raft_consensus.cc:348] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.911887 29197 raft_consensus.cc:374] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:27.911999 29197 raft_consensus.cc:729] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35768e019b0449e0abcbface2d505384, State: Initialized, Role: FOLLOWER
I20260808 15:42:27.912649 29199 raft_consensus.cc:348] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.912763 29197 consensus_queue.cc:260] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.913080 29199 raft_consensus.cc:374] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:27.913231 29199 raft_consensus.cc:729] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4a5fdfe8048568b89dae450b48d8e, State: Initialized, Role: FOLLOWER
I20260808 15:42:27.913367 29198 raft_consensus.cc:348] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.913841 29198 raft_consensus.cc:374] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:27.913988 29176 heartbeater.cc:503] Master 127.23.130.62:42953 was elected leader, sending a full tablet report...
I20260808 15:42:27.913985 29198 raft_consensus.cc:729] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a9b93970454abe91391cb7797fd3ef, State: Initialized, Role: FOLLOWER
I20260808 15:42:27.914018 29199 consensus_queue.cc:260] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.914808 29197 ts_tablet_manager.cc:1428] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260808 15:42:27.914806 29198 consensus_queue.cc:260] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:27.916575 29198 ts_tablet_manager.cc:1428] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260808 15:42:27.917546 29029 heartbeater.cc:503] Master 127.23.130.62:42953 was elected leader, sending a full tablet report...
I20260808 15:42:27.918164 29199 ts_tablet_manager.cc:1428] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260808 15:42:27.969750 28882 tablet.cc:2369] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:42:27.992722 29030 tablet.cc:2369] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:42:28.030877 29177 tablet.cc:2369] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:28.149431 29205 raft_consensus.cc:482] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:28.149818 29205 raft_consensus.cc:504] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:28.151611 29205 leader_election.cc:288] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dad4a5fdfe8048568b89dae450b48d8e (127.23.130.2:45577), 05a9b93970454abe91391cb7797fd3ef (127.23.130.1:39317)
I20260808 15:42:28.157946 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.152017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:45577 (local address 127.23.130.3:50305)
0808 15:42:28.152234 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:28.152256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:28.152403 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:28.153075 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:28.153080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:28.153109 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:28.153315 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:28.153327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.154823 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.154830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:28.156087 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:28.156098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.157209 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.157214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:28.157399 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.157424 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:42:28.157526 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:42:28.157656 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:42:28.157946 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.152085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:50305 (local address 127.23.130.2:45577)
0808 15:42:28.152267 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.152272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.152318 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:42:28.152612 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.152619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.152696 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.152850 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:28.153504 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.154658 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.156255 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.157041 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.157208 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.157289 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.157565 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.157695 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:42:28.158864 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.152472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:39317 (local address 127.23.130.3:55447)
0808 15:42:28.153557 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:28.153579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:28.153705 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:28.154170 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:28.154173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:28.154191 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:28.154564 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:28.154577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.155618 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.155624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:28.156962 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:28.156973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.158089 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.158095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:28.158283 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.158343 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:42:28.158476 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 15:42:28.158604 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":403,"threads_started":1}
I20260808 15:42:28.158893 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.152726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:55447 (local address 127.23.130.1:39317)
0808 15:42:28.152989 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.152995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.153675 (+ 680us) server_negotiation.cc:407] Connection header received
0808 15:42:28.153877 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.153883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.153967 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.154017 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:28.154728 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.155472 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.157124 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.157885 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.158070 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.158139 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.158511 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.158645 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260808 15:42:28.159209 28979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9275cfdc572b429382fce788a4ee4573" candidate_uuid: "35768e019b0449e0abcbface2d505384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4a5fdfe8048568b89dae450b48d8e" is_pre_election: true
I20260808 15:42:28.159816 28979 raft_consensus.cc:2454] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35768e019b0449e0abcbface2d505384 in term 0.
I20260808 15:42:28.159884 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9275cfdc572b429382fce788a4ee4573" candidate_uuid: "35768e019b0449e0abcbface2d505384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a9b93970454abe91391cb7797fd3ef" is_pre_election: true
I20260808 15:42:28.160494 28832 raft_consensus.cc:2454] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35768e019b0449e0abcbface2d505384 in term 0.
I20260808 15:42:28.160516 29054 leader_election.cc:302] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35768e019b0449e0abcbface2d505384, dad4a5fdfe8048568b89dae450b48d8e; no voters:
I20260808 15:42:28.160945 29205 raft_consensus.cc:2781] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:28.161118 29205 raft_consensus.cc:482] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:28.161257 29205 raft_consensus.cc:3037] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:28.163694 29205 raft_consensus.cc:504] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:28.164309 29205 leader_election.cc:288] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [CANDIDATE]: Term 1 election: Requested vote from peers dad4a5fdfe8048568b89dae450b48d8e (127.23.130.2:45577), 05a9b93970454abe91391cb7797fd3ef (127.23.130.1:39317)
I20260808 15:42:28.164822 28979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9275cfdc572b429382fce788a4ee4573" candidate_uuid: "35768e019b0449e0abcbface2d505384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4a5fdfe8048568b89dae450b48d8e"
I20260808 15:42:28.165035 28979 raft_consensus.cc:3037] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:28.164961 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9275cfdc572b429382fce788a4ee4573" candidate_uuid: "35768e019b0449e0abcbface2d505384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a9b93970454abe91391cb7797fd3ef"
I20260808 15:42:28.165218 28832 raft_consensus.cc:3037] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:28.167394 28979 raft_consensus.cc:2454] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35768e019b0449e0abcbface2d505384 in term 1.
I20260808 15:42:28.167816 28832 raft_consensus.cc:2454] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35768e019b0449e0abcbface2d505384 in term 1.
I20260808 15:42:28.167834 29054 leader_election.cc:302] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35768e019b0449e0abcbface2d505384, dad4a5fdfe8048568b89dae450b48d8e; no voters:
I20260808 15:42:28.168180 29205 raft_consensus.cc:2781] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:28.168660 29205 raft_consensus.cc:686] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 1 LEADER]: Becoming Leader. State: Replica: 35768e019b0449e0abcbface2d505384, State: Running, Role: LEADER
I20260808 15:42:28.169256 29205 consensus_queue.cc:237] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } }
I20260808 15:42:28.173913 28669 catalog_manager.cc:5495] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35768e019b0449e0abcbface2d505384 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "35768e019b0449e0abcbface2d505384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35768e019b0449e0abcbface2d505384" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 35499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:28.238708 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.233190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.23.130.3:35499)
0808 15:42:28.233407 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.233414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.233536 (+ 122us) server_negotiation.cc:407] Connection header received
0808 15:42:28.233786 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.233790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.233870 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.234028 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:28.234460 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.235170 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.236506 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.236909 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.237045 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.237555 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:28.238072 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:28.238249 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.238301 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.238382 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:42:28.247484 28832 raft_consensus.cc:1261] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 1 FOLLOWER]: Refusing update from remote peer 35768e019b0449e0abcbface2d505384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:28.247503 28979 raft_consensus.cc:1261] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 1 FOLLOWER]: Refusing update from remote peer 35768e019b0449e0abcbface2d505384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:28.248425 29205 consensus_queue.cc:1035] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad4a5fdfe8048568b89dae450b48d8e" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:28.248759 29209 consensus_queue.cc:1035] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a9b93970454abe91391cb7797fd3ef" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:28.260003 29213 mvcc.cc:205] Tried to move back new op lower bound from 7316290552810422272 to 7316290552505024512. Current Snapshot: MvccSnapshot[applied={T|T < 7316290552810422272}]
I20260808 15:42:28.265858 29215 mvcc.cc:205] Tried to move back new op lower bound from 7316290552810422272 to 7316290552505024512. Current Snapshot: MvccSnapshot[applied={T|T < 7316290552810422272}]
I20260808 15:42:28.274993 28671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:28.275250 28671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:28.278726 28671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f16e032796704dad8cf515a54f49a235: Sending DeleteTablet for 3 replicas of tablet 9275cfdc572b429382fce788a4ee4573
I20260808 15:42:28.279800 29107 tablet_service.cc:1514] Processing DeleteTablet for tablet 9275cfdc572b429382fce788a4ee4573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:28 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:49200
I20260808 15:42:28.279923 28958 tablet_service.cc:1514] Processing DeleteTablet for tablet 9275cfdc572b429382fce788a4ee4573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:28 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:47312
I20260808 15:42:28.280798 28811 tablet_service.cc:1514] Processing DeleteTablet for tablet 9275cfdc572b429382fce788a4ee4573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:28 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46600
I20260808 15:42:28.281215 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28740
I20260808 15:42:28.289592 29223 tablet_replica.cc:330] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: stopping tablet replica
I20260808 15:42:28.290526 29224 tablet_replica.cc:330] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: stopping tablet replica
I20260808 15:42:28.290951 29224 raft_consensus.cc:2229] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:28.291224 29223 raft_consensus.cc:2229] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:28.291965 29224 raft_consensus.cc:2258] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:28.292683 29225 tablet_replica.cc:330] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: stopping tablet replica
I20260808 15:42:28.293140 29225 raft_consensus.cc:2229] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:28.293218 29223 raft_consensus.cc:2258] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:28.293689 29225 raft_consensus.cc:2258] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:28.294973 29224 ts_tablet_manager.cc:1905] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:28.295783 29225 ts_tablet_manager.cc:1905] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:28.297462 29223 ts_tablet_manager.cc:1905] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:28.302357 29224 ts_tablet_manager.cc:1918] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:28.302610 29224 log.cc:1196] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/9275cfdc572b429382fce788a4ee4573
I20260808 15:42:28.303033 29225 ts_tablet_manager.cc:1918] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:28.303280 29225 log.cc:1196] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/9275cfdc572b429382fce788a4ee4573
I20260808 15:42:28.303293 29224 ts_tablet_manager.cc:1939] T 9275cfdc572b429382fce788a4ee4573 P dad4a5fdfe8048568b89dae450b48d8e: Deleting consensus metadata
I20260808 15:42:28.303870 29225 ts_tablet_manager.cc:1939] T 9275cfdc572b429382fce788a4ee4573 P 05a9b93970454abe91391cb7797fd3ef: Deleting consensus metadata
I20260808 15:42:28.304930 29223 ts_tablet_manager.cc:1918] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:28.305105 29223 log.cc:1196] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/9275cfdc572b429382fce788a4ee4573
I20260808 15:42:28.305445 28656 catalog_manager.cc:4841] TS dad4a5fdfe8048568b89dae450b48d8e (127.23.130.2:45577): tablet 9275cfdc572b429382fce788a4ee4573 (table test-table [id=68b5654f95164b6f92490435e6cdac28]) successfully deleted
I20260808 15:42:28.305675 29223 ts_tablet_manager.cc:1939] T 9275cfdc572b429382fce788a4ee4573 P 35768e019b0449e0abcbface2d505384: Deleting consensus metadata
I20260808 15:42:28.305812 28656 catalog_manager.cc:4841] TS 05a9b93970454abe91391cb7797fd3ef (127.23.130.1:39317): tablet 9275cfdc572b429382fce788a4ee4573 (table test-table [id=68b5654f95164b6f92490435e6cdac28]) successfully deleted
I20260808 15:42:28.307289 28658 catalog_manager.cc:4841] TS 35768e019b0449e0abcbface2d505384 (127.23.130.3:35499): tablet 9275cfdc572b429382fce788a4ee4573 (table test-table [id=68b5654f95164b6f92490435e6cdac28]) successfully deleted
Aug 08 15:42:28 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:28.318703 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.282601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.23.130.1:39317)
0808 15:42:28.282795 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.282801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.289238 (+ 6437us) server_negotiation.cc:407] Connection header received
0808 15:42:28.289592 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.289597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.289684 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.289761 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:28.292616 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.293802 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.306600 (+ 12798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.307309 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.307450 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.312000 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:28.312016 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:28.312036 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:28.312104 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:28.315807 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.316524 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.316534 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.316537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.316705 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.317069 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.317076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.317080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.317538 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:28.317890 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.318062 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.318289 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:28.319491 28876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:28.408027 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28740
I20260808 15:42:28.428090 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28887
Aug 08 15:42:28 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:28.446003 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.428928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.23.130.2:45577)
0808 15:42:28.429101 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.429106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.429126 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:42:28.429346 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.429351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.429415 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.429498 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:28.431362 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.432088 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.434011 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.434463 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.434571 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.439760 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:28.439776 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:28.439791 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:28.439835 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:28.443006 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.444009 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.444017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.444019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.444172 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.444575 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.444578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.444580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.444954 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:28.445166 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.445516 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.445680 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:42:28.446713 29022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:28.537166 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28887
I20260808 15:42:28.558460 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29035
Aug 08 15:42:28 dist-test-slave-gp6n krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:28.574795 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:28.559403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.23.130.3:35499)
0808 15:42:28.559612 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:28.559624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:28.559645 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:42:28.559709 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:28.559718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:28.559818 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:28.559940 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:28.561255 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.562056 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.563417 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:28.563911 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:28.564033 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:28.568467 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:28.568481 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:28.568496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:28.568532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:28.571786 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.572512 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.572525 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.572528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.572670 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:28.573076 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:28.573081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:28.573084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:28.573511 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:28.573843 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:28.574136 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:42:28.574386 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:42:28.575623 29171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:28.697885 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29035
I20260808 15:42:28.720642 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 28639
I20260808 15:42:28.721818 28706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:28.831996 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 28639
2026-08-08T15:42:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4602 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T15:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:28Z Disabled control of system clock
I20260808 15:42:28.908706 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44015
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:44015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:42:29.251787 29236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:29.252079 29236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:29.252251 29236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:29.259317 29236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:29.259449 29236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:29.259519 29236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:29.259595 29236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:29.259680 29236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:29.268690 29236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:44015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:29.269992 29236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:29.272071 29236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:29.285027 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:29.285089 29242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:29.285131 29241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:29.285965 29236 server_base.cc:1034] running on GCE node
I20260808 15:42:29.286865 29236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:29.288332 29236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:29.289577 29236 hybrid_clock.cc:648] HybridClock initialized: now 1786203749289539 us; error 53 us; skew 500 ppm
I20260808 15:42:29.289981 29236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:29.292786 29236 webserver.cc:460] Webserver started at http://127.23.130.62:33107/ using document root <none> and password file <none>
I20260808 15:42:29.293551 29236 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:29.293705 29236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:29.294073 29236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:29.297341 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "49eee7f81b5f4ed69dc6f892d6a34879"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "e22e18ab77491fea0a868eaa4554adcb"
server_key_iv: "509e873ee838de625a726962f4d05212"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.298336 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "49eee7f81b5f4ed69dc6f892d6a34879"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "e22e18ab77491fea0a868eaa4554adcb"
server_key_iv: "509e873ee838de625a726962f4d05212"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.303932 29236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:42:29.307768 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:29.309239 29236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:29.309468 29236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "49eee7f81b5f4ed69dc6f892d6a34879"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "e22e18ab77491fea0a868eaa4554adcb"
server_key_iv: "509e873ee838de625a726962f4d05212"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.309719 29236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:29.316010 29236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:29.316847 29236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:29.317181 29236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:29.336483 29236 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:44015
I20260808 15:42:29.336534 29306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:44015 every 8 connection(s)
I20260808 15:42:29.337950 29236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:29.339190 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29236
I20260808 15:42:29.339452 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:29.340173 24072 external_mini_cluster.cc:1428] Setting key c80432815d6335c020aca4806f7e87e1
I20260808 15:42:29.342708 29307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:29.349169 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:29.342294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.23.130.62:44015)
0808 15:42:29.343511 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:29.343523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:29.343566 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:42:29.345287 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:29.345312 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:29.345750 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:29.346120 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:29.347376 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:29.347382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:29.347395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:29.347412 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:29.347508 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:29.347525 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:29.347710 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:29.347875 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 15:42:29.348157 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":458,"threads_started":1}
I20260808 15:42:29.355897 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: Bootstrap starting.
I20260808 15:42:29.359321 29307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:29.360522 29307 log.cc:824] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:29.363364 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: No bootstrap required, opened a new log
I20260808 15:42:29.371258 29307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } }
I20260808 15:42:29.371611 29307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:29.371690 29307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49eee7f81b5f4ed69dc6f892d6a34879, State: Initialized, Role: FOLLOWER
I20260808 15:42:29.372267 29307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } }
I20260808 15:42:29.372494 29307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:29.372601 29307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:29.372726 29307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:29.375254 29307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } }
I20260808 15:42:29.375811 29307 leader_election.cc:302] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49eee7f81b5f4ed69dc6f892d6a34879; no voters:
I20260808 15:42:29.376406 29307 leader_election.cc:288] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:29.376557 29313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:29.377434 29313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [term 1 LEADER]: Becoming Leader. State: Replica: 49eee7f81b5f4ed69dc6f892d6a34879, State: Running, Role: LEADER
I20260808 15:42:29.378026 29313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } }
I20260808 15:42:29.378728 29307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:29.380967 29316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49eee7f81b5f4ed69dc6f892d6a34879. Latest consensus state: current_term: 1 leader_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } } }
I20260808 15:42:29.381323 29316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:29.383596 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eee7f81b5f4ed69dc6f892d6a34879" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44015 } } }
I20260808 15:42:29.384030 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:29.387394 29320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:29.392765 29320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:29.402765 29320 catalog_manager.cc:1347] Generated new cluster ID: 539f458d38124b94992d1a18f9860edf
I20260808 15:42:29.402915 29320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:29.434028 29320 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:29.434537 29320 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:29.439822 29320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: Generated new TSK 0
I20260808 15:42:29.440600 29320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:42:29.448872 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--builtin_ntp_servers=127.23.130.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:42:29.796443 29335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:29.796722 29335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:29.796881 29335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:29.804181 29335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:29.804275 29335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:29.804478 29335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:29.813796 29335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:29.815187 29335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:29.816990 29335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:29.829949 29341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:29.830076 29343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:29.830494 29345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:29.830928 29335 server_base.cc:1034] running on GCE node
I20260808 15:42:29.831490 29335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:29.832412 29335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:29.833635 29335 hybrid_clock.cc:648] HybridClock initialized: now 1786203749833594 us; error 57 us; skew 500 ppm
I20260808 15:42:29.833961 29335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:29.836246 29335 webserver.cc:460] Webserver started at http://127.23.130.1:38063/ using document root <none> and password file <none>
I20260808 15:42:29.836980 29335 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:29.837113 29335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:29.837433 29335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:29.840263 29335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "58f1f45db15d4a3481b59fa7500cf0c2"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "6ddb9a7d1fa9ef352ea719a0e2c7fc12"
server_key_iv: "8d36e1fae50cbabf938ec8186c5c51bf"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.841061 29335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "58f1f45db15d4a3481b59fa7500cf0c2"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "6ddb9a7d1fa9ef352ea719a0e2c7fc12"
server_key_iv: "8d36e1fae50cbabf938ec8186c5c51bf"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.846223 29335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:42:29.849946 29351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:29.851362 29335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:29.851547 29335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "58f1f45db15d4a3481b59fa7500cf0c2"
format_stamp: "Formatted at 2026-08-08 15:42:29 on dist-test-slave-gp6n"
server_key: "6ddb9a7d1fa9ef352ea719a0e2c7fc12"
server_key_iv: "8d36e1fae50cbabf938ec8186c5c51bf"
server_key_version: "encryptionkey@0"
I20260808 15:42:29.851778 29335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:29.857476 29335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:29.858259 29335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:29.858578 29335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:29.859488 29335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:29.860841 29335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:29.860971 29335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:29.861091 29335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:29.861181 29335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:29.894572 29335 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:35387
I20260808 15:42:29.894616 29492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:35387 every 8 connection(s)
I20260808 15:42:29.895947 29335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:29.903676 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29335
I20260808 15:42:29.903925 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:29.904768 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:29.898848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:59829 (local address 127.23.130.62:44015)
0808 15:42:29.899380 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:29.899387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:29.900643 (+ 1256us) server_negotiation.cc:407] Connection header received
0808 15:42:29.902294 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:29.902299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:29.902366 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:29.902540 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:29.903917 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:29.903924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:29.903928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:29.903959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:29.904021 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:29.904039 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:29.904193 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:29.904366 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 15:42:29.904437 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260808 15:42:29.904771 24072 external_mini_cluster.cc:1428] Setting key 47f1b0573583c51f048d338ac8edd638
I20260808 15:42:29.905529 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:29.899519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44015 (local address 127.23.130.1:59829)
0808 15:42:29.900472 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:29.900508 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:29.902017 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:29.902718 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:29.902726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:29.902817 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:29.903577 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:29.903579 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:29.903617 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:29.903699 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:29.904151 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:29.904159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:29.904164 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:42:29.904390 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 15:42:29.904577 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260808 15:42:29.907086 29493 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44015
I20260808 15:42:29.907380 29493 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:29.907670 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--builtin_ntp_servers=127.23.130.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:42:29.908021 29493 heartbeater.cc:511] Master 127.23.130.62:44015 requested a full tablet report, sending...
I20260808 15:42:29.910259 29266 ts_manager.cc:194] Registered new tserver with Master: 58f1f45db15d4a3481b59fa7500cf0c2 (127.23.130.1:35387)
W20260808 15:42:30.241154 29497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:30.241438 29497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:30.241528 29497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:30.248642 29497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:30.248797 29497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:30.248935 29497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:30.258021 29497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:30.259349 29497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:30.261075 29497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:30.274111 29502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:30.274075 29504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:30.274145 29507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:30.275105 29497 server_base.cc:1034] running on GCE node
I20260808 15:42:30.275764 29497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:30.276695 29497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:30.277968 29497 hybrid_clock.cc:648] HybridClock initialized: now 1786203750277909 us; error 47 us; skew 500 ppm
I20260808 15:42:30.278285 29497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:30.280646 29497 webserver.cc:460] Webserver started at http://127.23.130.2:33907/ using document root <none> and password file <none>
I20260808 15:42:30.281347 29497 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:30.281459 29497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:30.281795 29497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:30.284627 29497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "00906482270e4c038bebe44df78857a8"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "6244834b34374451b5489449a32d31d2"
server_key_iv: "0bc97abbccee501e154b8a6b6b747a46"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.285459 29497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "00906482270e4c038bebe44df78857a8"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "6244834b34374451b5489449a32d31d2"
server_key_iv: "0bc97abbccee501e154b8a6b6b747a46"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.290386 29497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:42:30.293637 29514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.295182 29497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:42:30.295464 29497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "00906482270e4c038bebe44df78857a8"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "6244834b34374451b5489449a32d31d2"
server_key_iv: "0bc97abbccee501e154b8a6b6b747a46"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.295665 29497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:30.301147 29497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:30.301884 29497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:30.302140 29497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:30.303071 29497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:30.304417 29497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:30.304529 29497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.304659 29497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:30.304736 29497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.338035 29497 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:35731
I20260808 15:42:30.338065 29638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:35731 every 8 connection(s)
I20260808 15:42:30.339366 29497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:30.340500 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29497
I20260808 15:42:30.340698 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:30.341233 24072 external_mini_cluster.cc:1428] Setting key 486ea9611e1d6e7b9f62be6389071bf8
I20260808 15:42:30.344830 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--builtin_ntp_servers=127.23.130.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:42:30.348805 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.342757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:52975 (local address 127.23.130.62:44015)
0808 15:42:30.342971 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.342977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.344505 (+ 1528us) server_negotiation.cc:407] Connection header received
0808 15:42:30.346340 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.346346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.346461 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.346561 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.348028 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.348034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.348040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.348066 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.348097 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.348109 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.348233 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.348430 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.348503 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:42:30.349858 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.343261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44015 (local address 127.23.130.2:52975)
0808 15:42:30.344341 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.344393 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.346070 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.346937 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.346945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.347037 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.347729 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.347731 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.347770 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.347851 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.348206 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.348218 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.348226 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.348522 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.348780 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":348,"threads_started":1}
I20260808 15:42:30.351255 29639 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44015
I20260808 15:42:30.351567 29639 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:30.352160 29639 heartbeater.cc:511] Master 127.23.130.62:44015 requested a full tablet report, sending...
I20260808 15:42:30.353566 29266 ts_manager.cc:194] Registered new tserver with Master: 00906482270e4c038bebe44df78857a8 (127.23.130.2:35731)
W20260808 15:42:30.675935 29642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:30.676213 29642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:30.676301 29642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:30.683818 29642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:30.683915 29642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:30.684023 29642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:30.693001 29642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:30.694167 29642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:30.695964 29642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:30.710074 29647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:30.710155 29648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:30.710129 29650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:30.711138 29642 server_base.cc:1034] running on GCE node
I20260808 15:42:30.711730 29642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:30.712663 29642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:30.713910 29642 hybrid_clock.cc:648] HybridClock initialized: now 1786203750713839 us; error 51 us; skew 500 ppm
I20260808 15:42:30.714241 29642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:30.716527 29642 webserver.cc:460] Webserver started at http://127.23.130.3:36163/ using document root <none> and password file <none>
I20260808 15:42:30.717223 29642 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:30.717355 29642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:30.717689 29642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:30.720597 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "59cdb0e72de544b386a0b50ff6cf6502"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "fa59acc3e88ba5a7f4e6ec75f0a9ca4b"
server_key_iv: "f99158f77f5d9e9a99ead7760e9f7733"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.721391 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "59cdb0e72de544b386a0b50ff6cf6502"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "fa59acc3e88ba5a7f4e6ec75f0a9ca4b"
server_key_iv: "f99158f77f5d9e9a99ead7760e9f7733"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.726363 29642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260808 15:42:30.729671 29656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.731037 29642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:30.731209 29642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "59cdb0e72de544b386a0b50ff6cf6502"
format_stamp: "Formatted at 2026-08-08 15:42:30 on dist-test-slave-gp6n"
server_key: "fa59acc3e88ba5a7f4e6ec75f0a9ca4b"
server_key_iv: "f99158f77f5d9e9a99ead7760e9f7733"
server_key_version: "encryptionkey@0"
I20260808 15:42:30.731431 29642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:30.736856 29642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:30.737627 29642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:30.737898 29642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:30.738852 29642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:30.740204 29642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:30.740346 29642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.740486 29642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:30.740583 29642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:30.774390 29642 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:37091
I20260808 15:42:30.774573 29780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:37091 every 8 connection(s)
I20260808 15:42:30.775822 29642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:30.781296 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29642
I20260808 15:42:30.781497 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:30.781930 24072 external_mini_cluster.cc:1428] Setting key d07386e9c2a18f8ddeccc65fda83e061
I20260808 15:42:30.784475 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.778516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:47001 (local address 127.23.130.62:44015)
0808 15:42:30.778703 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.778708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.780209 (+ 1501us) server_negotiation.cc:407] Connection header received
0808 15:42:30.781954 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.781969 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.782043 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.782184 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.783679 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.783685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.783690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.783708 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.783770 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.783786 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.783900 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.784100 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.784165 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:42:30.785445 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.779147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44015 (local address 127.23.130.3:47001)
0808 15:42:30.780052 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.780095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.781687 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.782481 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.782494 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.782621 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.783385 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.783387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.783424 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.783513 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.783905 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.783914 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.783921 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.784132 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.784342 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260808 15:42:30.786787 29781 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44015
I20260808 15:42:30.787070 29781 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:30.787602 29781 heartbeater.cc:511] Master 127.23.130.62:44015 requested a full tablet report, sending...
I20260808 15:42:30.788924 29266 ts_manager.cc:194] Registered new tserver with Master: 59cdb0e72de544b386a0b50ff6cf6502 (127.23.130.3:37091)
I20260808 15:42:30.796994 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:42:30.804011 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.801530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.23.130.62:44015)
0808 15:42:30.801736 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.801746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.802191 (+ 445us) server_negotiation.cc:407] Connection header received
0808 15:42:30.802321 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.802324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.802375 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.802519 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.803381 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.803385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.803388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.803402 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.803433 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.803445 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.803555 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.803659 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.803705 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:42:30.807840 29266 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:30.815068 29266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:30.836601 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.827185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.23.130.1:35387)
0808 15:42:30.828070 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.828084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.828941 (+ 857us) server_negotiation.cc:407] Connection header received
0808 15:42:30.829310 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.829323 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.829622 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.831120 (+ 1498us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.835316 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.835329 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.835349 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.835386 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.835459 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.835484 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.835616 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.835995 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.836125 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":1197,"threads_started":1}
I20260808 15:42:30.839206 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.827114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49524)
0808 15:42:30.828767 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.828817 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.829092 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.832463 (+ 3371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.832473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.832589 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.834926 (+ 2337us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.834932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.834986 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.835091 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.835734 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.835749 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.835760 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.837062 (+ 1302us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.837181 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":1143,"threads_started":1}
I20260808 15:42:30.842633 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.827820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:35731 (local address 127.0.0.1:43542)
0808 15:42:30.839632 (+ 11812us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.839655 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.839869 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.841059 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.841066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.841106 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.841700 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.841706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.841724 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.841769 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.842152 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.842159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.842164 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.842266 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.842322 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11673,"thread_start_us":12276,"threads_started":1}
I20260808 15:42:30.842762 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.828121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.23.130.2:35731)
0808 15:42:30.838612 (+ 10491us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.838625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.839773 (+ 1148us) server_negotiation.cc:407] Connection header received
0808 15:42:30.840060 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.840072 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.840333 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.840434 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.841915 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.841925 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.841939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.841968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.842034 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.842055 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.842227 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.842287 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.842382 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10263,"thread_start_us":10506,"threads_started":1}
I20260808 15:42:30.843868 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.827718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37091 (local address 127.0.0.1:48774)
0808 15:42:30.839453 (+ 11735us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.839471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.839589 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.841128 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.841132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.841149 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.841864 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.841867 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.841883 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.841919 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.843283 (+ 1364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.843288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.843293 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.843412 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.843461 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11612,"thread_start_us":11449,"threads_started":1}
I20260808 15:42:30.843987 29417 tablet_service.cc:1467] Processing CreateTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 (DEFAULT_TABLE table=test-table [id=a86f1b76f04546aebc6626a8eb914608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:30.845173 29417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654981e0c1894fe197ab2fbb3e8d5c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:30.845927 29565 tablet_service.cc:1467] Processing CreateTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 (DEFAULT_TABLE table=test-table [id=a86f1b76f04546aebc6626a8eb914608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:30.847229 29565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654981e0c1894fe197ab2fbb3e8d5c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:30.847391 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.828257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.23.130.3:37091)
0808 15:42:30.840630 (+ 12373us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.840638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.840671 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:42:30.840743 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.840752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.840945 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.841535 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.843007 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.843018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.843033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.843057 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.843124 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.843151 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.846826 (+ 3675us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.846911 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.847006 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12209,"thread_start_us":12512,"threads_started":1}
I20260808 15:42:30.850940 29709 tablet_service.cc:1467] Processing CreateTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 (DEFAULT_TABLE table=test-table [id=a86f1b76f04546aebc6626a8eb914608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:30.852294 29709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654981e0c1894fe197ab2fbb3e8d5c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:30.858963 29800 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Bootstrap starting.
I20260808 15:42:30.861035 29801 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Bootstrap starting.
I20260808 15:42:30.862638 29800 tablet_bootstrap.cc:654] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:30.863958 29800 log.cc:824] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:30.864786 29801 tablet_bootstrap.cc:654] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:30.866042 29801 log.cc:824] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:30.866684 29804 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Bootstrap starting.
I20260808 15:42:30.867352 29800 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: No bootstrap required, opened a new log
I20260808 15:42:30.867645 29800 ts_tablet_manager.cc:1397] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260808 15:42:30.869907 29801 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: No bootstrap required, opened a new log
I20260808 15:42:30.870182 29801 ts_tablet_manager.cc:1397] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:42:30.870527 29804 tablet_bootstrap.cc:654] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:30.871692 29804 log.cc:824] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:30.874599 29804 tablet_bootstrap.cc:492] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: No bootstrap required, opened a new log
I20260808 15:42:30.874842 29804 ts_tablet_manager.cc:1397] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260808 15:42:30.875590 29800 raft_consensus.cc:348] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.876142 29800 raft_consensus.cc:374] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:30.876299 29800 raft_consensus.cc:729] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00906482270e4c038bebe44df78857a8, State: Initialized, Role: FOLLOWER
I20260808 15:42:30.876996 29800 consensus_queue.cc:260] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.878027 29639 heartbeater.cc:503] Master 127.23.130.62:44015 was elected leader, sending a full tablet report...
I20260808 15:42:30.879988 29800 ts_tablet_manager.cc:1428] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260808 15:42:30.880329 29801 raft_consensus.cc:348] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.880731 29801 raft_consensus.cc:374] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:30.880831 29801 raft_consensus.cc:729] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f1f45db15d4a3481b59fa7500cf0c2, State: Initialized, Role: FOLLOWER
I20260808 15:42:30.881429 29801 consensus_queue.cc:260] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.883055 29801 ts_tablet_manager.cc:1428] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 15:42:30.883142 29493 heartbeater.cc:503] Master 127.23.130.62:44015 was elected leader, sending a full tablet report...
I20260808 15:42:30.884702 29804 raft_consensus.cc:348] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.885035 29804 raft_consensus.cc:374] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:30.885187 29804 raft_consensus.cc:729] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cdb0e72de544b386a0b50ff6cf6502, State: Initialized, Role: FOLLOWER
I20260808 15:42:30.885777 29804 consensus_queue.cc:260] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.886921 29781 heartbeater.cc:503] Master 127.23.130.62:44015 was elected leader, sending a full tablet report...
I20260808 15:42:30.887435 29804 ts_tablet_manager.cc:1428] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260808 15:42:30.898249 29494 tablet.cc:2369] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:30.927917 29808 raft_consensus.cc:482] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:30.928292 29808 raft_consensus.cc:504] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.930071 29808 leader_election.cc:288] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f1f45db15d4a3481b59fa7500cf0c2 (127.23.130.1:35387), 59cdb0e72de544b386a0b50ff6cf6502 (127.23.130.3:37091)
I20260808 15:42:30.933985 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.930688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:42027 (local address 127.23.130.3:37091)
0808 15:42:30.930889 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.930896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.931793 (+ 897us) server_negotiation.cc:407] Connection header received
0808 15:42:30.931980 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.931985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.932047 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.932156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.933328 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.933334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.933339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.933358 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.933396 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.933412 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.933566 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.933649 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.933712 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:42:30.934006 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.930682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:51627 (local address 127.23.130.1:35387)
0808 15:42:30.930891 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.930897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.931809 (+ 912us) server_negotiation.cc:407] Connection header received
0808 15:42:30.931977 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.931983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.932047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.932156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.933328 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.933334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.933339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.933358 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.933396 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.933412 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.933546 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.933651 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.933711 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 15:42:30.934484 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.930626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.23.130.2:51627)
0808 15:42:30.931707 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.931724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.931824 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.932200 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.932205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.932230 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.933097 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.933102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.933122 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.933173 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.933528 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.933532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.933537 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.933709 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.933764 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":472,"threads_started":1}
I20260808 15:42:30.934569 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.930395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37091 (local address 127.23.130.2:42027)
0808 15:42:30.931650 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:30.931670 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:30.931865 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:30.932200 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:30.932205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:30.932230 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:30.933124 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:42:30.933127 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:30.933138 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:42:30.933173 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:30.933532 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:30.933536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:30.933540 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 15:42:30.933698 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 15:42:30.933803 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":22,"spinlock_wait_cycles":34560,"thread_start_us":689,"threads_started":1}
I20260808 15:42:30.935681 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654981e0c1894fe197ab2fbb3e8d5c92" candidate_uuid: "00906482270e4c038bebe44df78857a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" is_pre_election: true
I20260808 15:42:30.935680 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654981e0c1894fe197ab2fbb3e8d5c92" candidate_uuid: "00906482270e4c038bebe44df78857a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cdb0e72de544b386a0b50ff6cf6502" is_pre_election: true
I20260808 15:42:30.936265 29732 raft_consensus.cc:2454] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00906482270e4c038bebe44df78857a8 in term 0.
I20260808 15:42:30.936264 29440 raft_consensus.cc:2454] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00906482270e4c038bebe44df78857a8 in term 0.
I20260808 15:42:30.936992 29515 leader_election.cc:302] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00906482270e4c038bebe44df78857a8, 58f1f45db15d4a3481b59fa7500cf0c2, 59cdb0e72de544b386a0b50ff6cf6502; no voters:
I20260808 15:42:30.937482 29808 raft_consensus.cc:2781] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:30.937667 29808 raft_consensus.cc:482] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:30.937817 29808 raft_consensus.cc:3037] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:30.940477 29808 raft_consensus.cc:504] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.941128 29808 leader_election.cc:288] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [CANDIDATE]: Term 1 election: Requested vote from peers 58f1f45db15d4a3481b59fa7500cf0c2 (127.23.130.1:35387), 59cdb0e72de544b386a0b50ff6cf6502 (127.23.130.3:37091)
I20260808 15:42:30.941597 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654981e0c1894fe197ab2fbb3e8d5c92" candidate_uuid: "00906482270e4c038bebe44df78857a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f1f45db15d4a3481b59fa7500cf0c2"
I20260808 15:42:30.941694 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654981e0c1894fe197ab2fbb3e8d5c92" candidate_uuid: "00906482270e4c038bebe44df78857a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cdb0e72de544b386a0b50ff6cf6502"
I20260808 15:42:30.941785 29440 raft_consensus.cc:3037] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:30.941969 29732 raft_consensus.cc:3037] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:30.944115 29440 raft_consensus.cc:2454] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00906482270e4c038bebe44df78857a8 in term 1.
I20260808 15:42:30.944490 29515 leader_election.cc:302] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00906482270e4c038bebe44df78857a8, 58f1f45db15d4a3481b59fa7500cf0c2; no voters:
I20260808 15:42:30.944626 29732 raft_consensus.cc:2454] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00906482270e4c038bebe44df78857a8 in term 1.
I20260808 15:42:30.944778 29808 raft_consensus.cc:2781] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:30.945308 29808 raft_consensus.cc:686] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 1 LEADER]: Becoming Leader. State: Replica: 00906482270e4c038bebe44df78857a8, State: Running, Role: LEADER
I20260808 15:42:30.945822 29808 consensus_queue.cc:237] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } }
I20260808 15:42:30.951042 29266 catalog_manager.cc:5495] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00906482270e4c038bebe44df78857a8 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "00906482270e4c038bebe44df78857a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00906482270e4c038bebe44df78857a8" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260808 15:42:30.989682 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:30.987760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.23.130.2:35731)
0808 15:42:30.987972 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:30.987976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:30.988022 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:42:30.988112 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:30.988117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:30.988211 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:30.988293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:30.989167 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:30.989172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:30.989175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:30.989189 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:30.989222 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:30.989234 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:30.989384 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:30.989446 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 15:42:30.989493 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:42:30.998267 29732 raft_consensus.cc:1261] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 1 FOLLOWER]: Refusing update from remote peer 00906482270e4c038bebe44df78857a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:30.998334 29440 raft_consensus.cc:1261] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 1 FOLLOWER]: Refusing update from remote peer 00906482270e4c038bebe44df78857a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:30.999459 29813 consensus_queue.cc:1035] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cdb0e72de544b386a0b50ff6cf6502" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:30.999862 29808 consensus_queue.cc:1035] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f1f45db15d4a3481b59fa7500cf0c2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:31.012949 29817 mvcc.cc:205] Tried to move back new op lower bound from 7316290564078313472 to 7316290563878993920. Current Snapshot: MvccSnapshot[applied={T|T < 7316290564078313472}]
I20260808 15:42:31.014778 29820 mvcc.cc:205] Tried to move back new op lower bound from 7316290564078313472 to 7316290563878993920. Current Snapshot: MvccSnapshot[applied={T|T < 7316290564078313472}]
I20260808 15:42:31.029230 29267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:31.029572 29267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:31.033201 29267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49eee7f81b5f4ed69dc6f892d6a34879: Sending DeleteTablet for 3 replicas of tablet 654981e0c1894fe197ab2fbb3e8d5c92
I20260808 15:42:31.034384 29709 tablet_service.cc:1514] Processing DeleteTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:31 UTC) from {username='slave'} at 127.0.0.1:48774
I20260808 15:42:31.034396 29417 tablet_service.cc:1514] Processing DeleteTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:31 UTC) from {username='slave'} at 127.0.0.1:49524
I20260808 15:42:31.034647 29565 tablet_service.cc:1514] Processing DeleteTablet for tablet 654981e0c1894fe197ab2fbb3e8d5c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:31 UTC) from {username='slave'} at 127.0.0.1:43542
I20260808 15:42:31.036791 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29335
I20260808 15:42:31.046429 29831 tablet_replica.cc:330] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: stopping tablet replica
I20260808 15:42:31.046993 29831 raft_consensus.cc:2229] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:31.047084 29829 tablet_replica.cc:330] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: stopping tablet replica
I20260808 15:42:31.047483 29829 raft_consensus.cc:2229] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:31.047735 29831 raft_consensus.cc:2258] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:31.048028 29829 raft_consensus.cc:2258] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:31.048596 29830 tablet_replica.cc:330] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: stopping tablet replica
I20260808 15:42:31.048933 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.038183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.23.130.1:35387)
0808 15:42:31.038357 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:31.038363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:31.043296 (+ 4933us) server_negotiation.cc:407] Connection header received
0808 15:42:31.043537 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:31.043543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:31.043652 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:31.043765 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:31.047727 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:31.047734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:31.047739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:31.047761 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:31.047803 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:31.047820 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:31.047947 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:31.048554 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 15:42:31.048627 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:42:31.048995 29830 raft_consensus.cc:2229] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:31.049500 29830 raft_consensus.cc:2258] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:31.050552 29488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:31.055487 29831 ts_tablet_manager.cc:1905] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:31.060890 29831 ts_tablet_manager.cc:1918] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:31.061085 29831 log.cc:1196] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/654981e0c1894fe197ab2fbb3e8d5c92
I20260808 15:42:31.061692 29831 ts_tablet_manager.cc:1939] T 654981e0c1894fe197ab2fbb3e8d5c92 P 00906482270e4c038bebe44df78857a8: Deleting consensus metadata
I20260808 15:42:31.063539 29830 ts_tablet_manager.cc:1905] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:31.063817 29253 catalog_manager.cc:4841] TS 00906482270e4c038bebe44df78857a8 (127.23.130.2:35731): tablet 654981e0c1894fe197ab2fbb3e8d5c92 (table test-table [id=a86f1b76f04546aebc6626a8eb914608]) successfully deleted
I20260808 15:42:31.068423 29830 ts_tablet_manager.cc:1918] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:31.068600 29830 log.cc:1196] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/654981e0c1894fe197ab2fbb3e8d5c92
I20260808 15:42:31.069079 29830 ts_tablet_manager.cc:1939] T 654981e0c1894fe197ab2fbb3e8d5c92 P 59cdb0e72de544b386a0b50ff6cf6502: Deleting consensus metadata
I20260808 15:42:31.071031 29252 catalog_manager.cc:4841] TS 59cdb0e72de544b386a0b50ff6cf6502 (127.23.130.3:37091): tablet 654981e0c1894fe197ab2fbb3e8d5c92 (table test-table [id=a86f1b76f04546aebc6626a8eb914608]) successfully deleted
I20260808 15:42:31.050843 29829 ts_tablet_manager.cc:1905] T 654981e0c1894fe197ab2fbb3e8d5c92 P 58f1f45db15d4a3481b59fa7500cf0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:31.183717 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29335
W20260808 15:42:31.206682 29251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:35387 (error 108)
I20260808 15:42:31.207459 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29497
I20260808 15:42:31.209627 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.209091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49536)
0808 15:42:31.209285 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.209441 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 15:42:31.209981 29251 catalog_manager.cc:4584] TS 58f1f45db15d4a3481b59fa7500cf0c2 (127.23.130.1:35387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 654981e0c1894fe197ab2fbb3e8d5c92: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
I20260808 15:42:31.211171 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.208677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.23.130.2:35731)
0808 15:42:31.208856 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:31.208862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:31.208882 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:42:31.208977 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:31.208983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:31.209055 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:31.209195 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:31.210385 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:31.210392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:31.210397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:31.210441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:31.210486 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:31.210505 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:31.210644 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:31.210777 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 15:42:31.210853 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:42:31.211843 29633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:31.264005 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.263397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49552)
0808 15:42:31.263666 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.263780 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 15:42:31.265098 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.264710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49562)
0808 15:42:31.264879 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.264966 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:42:31.324868 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29497
I20260808 15:42:31.326879 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.325173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49578)
0808 15:42:31.326550 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.326667 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1186}
I20260808 15:42:31.328117 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.327699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:49586)
0808 15:42:31.327842 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.327913 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:42:31.342860 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29642
I20260808 15:42:31.345811 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.343718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.23.130.3:37091)
0808 15:42:31.343897 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:31.343903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:31.343924 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:42:31.344101 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:31.344109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:31.344202 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:31.344309 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:31.345202 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:31.345208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:31.345213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:31.345230 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:31.345267 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:31.345282 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:31.345407 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:31.345460 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 15:42:31.345515 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:42:31.346517 29776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:31.408846 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.408276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:38722)
0808 15:42:31.408531 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.408648 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 15:42:31.410091 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:31.409649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35387 (local address 127.0.0.1:38732)
0808 15:42:31.409894 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:31.409970 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 15:42:31.435467 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29642
I20260808 15:42:31.454126 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29236
I20260808 15:42:31.455513 29301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:31.561424 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29236
2026-08-08T15:42:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2728 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 15:42:31.607262 24072 mini_oidc.cc:152] Starting JWKS server
I20260808 15:42:31.610216 24072 webserver.cc:460] Webserver started at https://127.0.0.1:33911/ using document root <none> and password file <none>
I20260808 15:42:31.610658 24072 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:42:31.611399 24072 webserver.cc:460] Webserver started at http://127.0.0.1:45167/ using document root <none> and password file <none>
2026-08-08T15:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:31Z Disabled control of system clock
I20260808 15:42:31.645041 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:40187
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:40187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33911/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/testchainca.cert with env {}
W20260808 15:42:31.973909 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:31.974255 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:31.974455 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:31.981606 29844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:31.981761 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:31.981843 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:31.981951 29844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:31.982038 29844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:42:31.982125 29844 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:42:31.982204 29844 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33911/jwks.json
I20260808 15:42:31.992314 29844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:40187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33911/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:31.993686 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:31.995074 29844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:32.007129 29849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:32.007172 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:32.007215 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:32.008159 29844 server_base.cc:1034] running on GCE node
I20260808 15:42:32.008955 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:32.010370 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:32.011641 29844 hybrid_clock.cc:648] HybridClock initialized: now 1786203752011584 us; error 55 us; skew 500 ppm
I20260808 15:42:32.012061 29844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:32.014683 29844 webserver.cc:460] Webserver started at http://127.23.130.62:43045/ using document root <none> and password file <none>
I20260808 15:42:32.015381 29844 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:32.015512 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:32.015847 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:32.018858 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "fb329b0a6aa34a038177284b91fc4c0d"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "c82356e94444b9de9f999f7f4ccca881"
server_key_iv: "3c3f4374302dc7f18164694d817a5699"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.019687 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "fb329b0a6aa34a038177284b91fc4c0d"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "c82356e94444b9de9f999f7f4ccca881"
server_key_iv: "3c3f4374302dc7f18164694d817a5699"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.024997 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:32.028395 29859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:32.029788 29844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:32.029965 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "fb329b0a6aa34a038177284b91fc4c0d"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "c82356e94444b9de9f999f7f4ccca881"
server_key_iv: "3c3f4374302dc7f18164694d817a5699"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.030174 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:32.045512 29844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:32.046458 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:32.046802 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:32.066128 29844 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:40187
I20260808 15:42:32.066148 29917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:40187 every 8 connection(s)
I20260808 15:42:32.067559 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:32.071249 29919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:32.075635 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29844
I20260808 15:42:32.075822 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:32.076498 24072 external_mini_cluster.cc:1428] Setting key e2097cc36e6e93f4b5b3b55566e682ab
I20260808 15:42:32.082366 29919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d: Bootstrap starting.
I20260808 15:42:32.085367 29919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:32.086452 29919 log.cc:824] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:32.089711 29919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d: No bootstrap required, opened a new log
I20260808 15:42:32.090307 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:32.078492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.23.130.62:40187)
0808 15:42:32.079550 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:32.079561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:32.079603 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:42:32.081363 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:32.081386 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:32.081833 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:32.082227 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:32.082849 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.084158 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.085530 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.086212 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.086615 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.087975 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:32.087983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:32.087996 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:32.088503 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:32.088578 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:32.088599 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:32.088685 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:32.088989 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 15:42:32.089329 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":362,"threads_started":1}
I20260808 15:42:32.094024 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:42:32.099452 29919 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } }
I20260808 15:42:32.099787 29919 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:32.099874 29919 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb329b0a6aa34a038177284b91fc4c0d, State: Initialized, Role: FOLLOWER
I20260808 15:42:32.100441 29919 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } }
I20260808 15:42:32.100654 29919 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:32.100759 29919 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:32.100894 29919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:32.103266 29919 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } }
I20260808 15:42:32.103735 29919 leader_election.cc:302] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb329b0a6aa34a038177284b91fc4c0d; no voters:
I20260808 15:42:32.104390 29919 leader_election.cc:288] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:32.104565 29925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:32.105361 29925 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [term 1 LEADER]: Becoming Leader. State: Replica: fb329b0a6aa34a038177284b91fc4c0d, State: Running, Role: LEADER
I20260808 15:42:32.105918 29925 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } }
I20260808 15:42:32.106717 29919 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:32.109361 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb329b0a6aa34a038177284b91fc4c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } } }
I20260808 15:42:32.109331 29928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb329b0a6aa34a038177284b91fc4c0d. Latest consensus state: current_term: 1 leader_uuid: "fb329b0a6aa34a038177284b91fc4c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb329b0a6aa34a038177284b91fc4c0d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 40187 } } }
I20260808 15:42:32.110416 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:32.110656 29928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:32.115186 29939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:32.122018 29939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:32.131944 29939 catalog_manager.cc:1347] Generated new cluster ID: b6b82646f9b3475994313d4c650f8506
I20260808 15:42:32.132079 29939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:32.157212 29939 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:32.158735 29939 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:32.173451 29939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb329b0a6aa34a038177284b91fc4c0d: Generated new TSK 0
I20260808 15:42:32.174449 29939 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:42:32.214849 29958 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 15:42:32.215117 29958 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:42:32.216718 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:32.190674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.23.130.62:40187)
0808 15:42:32.190965 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:32.190976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:32.191695 (+ 719us) server_negotiation.cc:407] Connection header received
0808 15:42:32.192175 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:32.192182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:32.192304 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:32.192476 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:42:32.193044 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.194346 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.195938 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.196743 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.196879 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.197904 (+ 1025us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:42:32.215784 (+ 17880us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:42:32.216295 (+ 511us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 15:42:32.216879 29922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 15:42:32.219861 29957 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:32.190559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:40187 (local address 127.0.0.1:45176)
0808 15:42:32.191565 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:32.191583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:32.191848 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:32.192509 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:32.192513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:32.192525 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:42:32.192884 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:32.192892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.194535 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.194539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:32.195773 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:32.195786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.197058 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.197065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:32.197307 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.197393 (+ 86us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:42:32.216272 (+ 18879us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:42:32.217514 (+ 1242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.130.62:40187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":673,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.130.62:40187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 15:42:32.222426 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29844
I20260808 15:42:32.224045 29913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:32.320881 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29844
2026-08-08T15:42:32Z chronyd exiting
I20260808 15:42:32.373317 24072 test_util.cc:175] -----------------------------------------------
I20260808 15:42:32.373474 24072 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786203706771851-24072-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (772 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 15:42:32.377343 24072 mini_oidc.cc:152] Starting JWKS server
I20260808 15:42:32.379302 24072 webserver.cc:460] Webserver started at https://127.0.0.1:43385/ using document root <none> and password file <none>
I20260808 15:42:32.379552 24072 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:42:32.380184 24072 webserver.cc:460] Webserver started at http://127.0.0.1:46565/ using document root <none> and password file <none>
2026-08-08T15:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:32Z Disabled control of system clock
I20260808 15:42:32.412472 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33675
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:33675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/testchainca.cert with env {}
W20260808 15:42:32.736546 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:32.736894 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:32.737068 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:32.744197 29966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:32.744339 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:32.744429 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:32.744535 29966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:32.744622 29966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:42:32.744709 29966 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:42:32.744788 29966 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43385/jwks.json
I20260808 15:42:32.754721 29966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:33675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:32.756062 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:32.757400 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:32.769424 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:32.769464 29975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:32.769495 29973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:32.770206 29966 server_base.cc:1034] running on GCE node
I20260808 15:42:32.770988 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:32.772392 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:32.773630 29966 hybrid_clock.cc:648] HybridClock initialized: now 1786203752773584 us; error 44 us; skew 500 ppm
I20260808 15:42:32.773976 29966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:32.780135 29966 webserver.cc:460] Webserver started at http://127.23.130.62:42281/ using document root <none> and password file <none>
I20260808 15:42:32.780934 29966 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:32.781075 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:32.781461 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:32.784332 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "2119bb8de4774cb79518cb4cddea8b12"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "b94f6d0955569913542cd4e4b7a9747c"
server_key_iv: "39dd676ab188097d31b84b16739791bd"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.785123 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "2119bb8de4774cb79518cb4cddea8b12"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "b94f6d0955569913542cd4e4b7a9747c"
server_key_iv: "39dd676ab188097d31b84b16739791bd"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.790135 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:42:32.793490 29981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:32.794939 29966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:32.795130 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "2119bb8de4774cb79518cb4cddea8b12"
format_stamp: "Formatted at 2026-08-08 15:42:32 on dist-test-slave-gp6n"
server_key: "b94f6d0955569913542cd4e4b7a9747c"
server_key_iv: "39dd676ab188097d31b84b16739791bd"
server_key_version: "encryptionkey@0"
I20260808 15:42:32.795351 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:32.825309 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:32.826229 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:32.826601 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:32.845913 29966 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:33675
I20260808 15:42:32.845924 30034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:33675 every 8 connection(s)
I20260808 15:42:32.847421 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:32.851250 30035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:32.852378 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 29966
I20260808 15:42:32.852609 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:32.853502 24072 external_mini_cluster.cc:1428] Setting key 936547237f7cb3397e06fece9d835e56
I20260808 15:42:32.865483 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12: Bootstrap starting.
I20260808 15:42:32.867038 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:32.855543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.23.130.62:33675)
0808 15:42:32.856615 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:32.856628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:32.856672 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:42:32.858198 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:32.858224 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:32.858762 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:32.859106 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:32.859639 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.860859 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.862698 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.863379 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.863518 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.864722 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:32.864729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:32.864741 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:32.865254 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:32.865327 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:32.865347 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:32.865430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:32.865720 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 15:42:32.866044 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":414,"threads_started":1}
I20260808 15:42:32.868683 30035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:32.869860 30035 log.cc:824] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:32.870810 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:42:32.872658 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12: No bootstrap required, opened a new log
I20260808 15:42:32.880625 30035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } }
I20260808 15:42:32.881042 30035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:32.881148 30035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2119bb8de4774cb79518cb4cddea8b12, State: Initialized, Role: FOLLOWER
I20260808 15:42:32.881692 30035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } }
I20260808 15:42:32.881945 30035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:32.882087 30035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:32.882263 30035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:32.884832 30035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } }
I20260808 15:42:32.885322 30035 leader_election.cc:302] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2119bb8de4774cb79518cb4cddea8b12; no voters:
I20260808 15:42:32.885888 30035 leader_election.cc:288] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:32.886015 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:32.886749 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [term 1 LEADER]: Becoming Leader. State: Replica: 2119bb8de4774cb79518cb4cddea8b12, State: Running, Role: LEADER
I20260808 15:42:32.887250 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } }
I20260808 15:42:32.887931 30035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:32.890165 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2119bb8de4774cb79518cb4cddea8b12. Latest consensus state: current_term: 1 leader_uuid: "2119bb8de4774cb79518cb4cddea8b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } } }
I20260808 15:42:32.890508 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2119bb8de4774cb79518cb4cddea8b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119bb8de4774cb79518cb4cddea8b12" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33675 } } }
I20260808 15:42:32.890766 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:32.890846 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:32.892820 30050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:32.897795 30050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:32.908730 30050 catalog_manager.cc:1347] Generated new cluster ID: 345d129c95064d818728c397587fa5e6
I20260808 15:42:32.908849 30050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:32.919328 30050 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:32.920600 30050 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:32.931354 30050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2119bb8de4774cb79518cb4cddea8b12: Generated new TSK 0
I20260808 15:42:32.932483 30050 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:42:32.970798 30069 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 15:42:32.970928 30069 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:42:32.972225 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:32.951761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.23.130.62:33675)
0808 15:42:32.951965 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:32.951972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:32.952359 (+ 387us) server_negotiation.cc:407] Connection header received
0808 15:42:32.952584 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:32.952591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:32.952695 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:32.952864 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:42:32.953311 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.954493 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.955852 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.956552 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.956679 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.957160 (+ 481us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:42:32.971510 (+ 14350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:42:32.971894 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:42:32.972242 30068 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:32.951616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33675 (local address 127.0.0.1:53618)
0808 15:42:32.952252 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:32.952271 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:32.952399 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:32.952874 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:32.952878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:32.952887 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:42:32.953155 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:32.953162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.954620 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.954624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:32.955716 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:32.955726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:32.956730 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:32.956734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:32.956865 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:32.956895 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:42:32.971753 (+ 14858us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:42:32.971951 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.130.62:33675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":347,"threads_started":1}
W20260808 15:42:32.972352 30038 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260808 15:42:32.973404 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 29966
I20260808 15:42:32.974702 30030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:33.068385 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 29966
2026-08-08T15:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 15:42:33.126597 24072 mini_oidc.cc:152] Starting JWKS server
I20260808 15:42:33.128480 24072 webserver.cc:460] Webserver started at https://127.0.0.1:37157/ using document root <none> and password file <none>
I20260808 15:42:33.128665 24072 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:42:33.129287 24072 webserver.cc:460] Webserver started at http://127.0.0.1:42895/ using document root <none> and password file <none>
2026-08-08T15:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:33Z Disabled control of system clock
I20260808 15:42:33.161867 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43837
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:43837
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:42:33.493281 30078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:33.493584 30078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:33.493785 30078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:33.500898 30078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:33.501025 30078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:33.501120 30078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:33.501227 30078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:33.501310 30078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:42:33.501411 30078 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:42:33.501494 30078 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37157/jwks.json
I20260808 15:42:33.510543 30078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:43837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:33.511793 30078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:33.513571 30078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:33.525831 30083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:33.525782 30084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:33.525869 30087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:33.526647 30078 server_base.cc:1034] running on GCE node
I20260808 15:42:33.527400 30078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:33.528730 30078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:33.529934 30078 hybrid_clock.cc:648] HybridClock initialized: now 1786203753529896 us; error 42 us; skew 500 ppm
I20260808 15:42:33.530344 30078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:33.532729 30078 webserver.cc:460] Webserver started at http://127.23.130.62:34633/ using document root <none> and password file <none>
I20260808 15:42:33.533424 30078 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:33.533542 30078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:33.533879 30078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:33.536718 30078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "f96d03f197734d27a7955a93766fad5c"
format_stamp: "Formatted at 2026-08-08 15:42:33 on dist-test-slave-gp6n"
server_key: "faa92ddbb0128ce024ce630769fdc69e"
server_key_iv: "3db4e5fd0daf74aa189ae69785e61a62"
server_key_version: "encryptionkey@0"
I20260808 15:42:33.537514 30078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "f96d03f197734d27a7955a93766fad5c"
format_stamp: "Formatted at 2026-08-08 15:42:33 on dist-test-slave-gp6n"
server_key: "faa92ddbb0128ce024ce630769fdc69e"
server_key_iv: "3db4e5fd0daf74aa189ae69785e61a62"
server_key_version: "encryptionkey@0"
I20260808 15:42:33.542519 30078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:42:33.545707 30094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:33.547151 30078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:33.547336 30078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "f96d03f197734d27a7955a93766fad5c"
format_stamp: "Formatted at 2026-08-08 15:42:33 on dist-test-slave-gp6n"
server_key: "faa92ddbb0128ce024ce630769fdc69e"
server_key_iv: "3db4e5fd0daf74aa189ae69785e61a62"
server_key_version: "encryptionkey@0"
I20260808 15:42:33.547554 30078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:33.560274 30078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:33.561025 30078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:33.561359 30078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:33.579905 30078 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:43837
I20260808 15:42:33.579941 30147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:43837 every 8 connection(s)
I20260808 15:42:33.581163 30078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:33.581941 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30078
I20260808 15:42:33.582170 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:33.582923 24072 external_mini_cluster.cc:1428] Setting key d08307f19a38a6ca0ee4492d43d7ecb4
I20260808 15:42:33.586042 30148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:33.596581 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:33.584570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.23.130.62:43837)
0808 15:42:33.585539 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:33.585549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:33.585591 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:42:33.587261 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:33.587281 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:33.587694 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:33.588020 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:33.588627 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.589945 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.591881 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.592575 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.592711 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:33.594005 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:33.594013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:33.594027 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:33.594678 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:33.594765 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:33.594788 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:33.594917 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:33.595244 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 15:42:33.595578 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":383,"threads_started":1}
I20260808 15:42:33.598207 30148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c: Bootstrap starting.
I20260808 15:42:33.600692 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:42:33.601737 30148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:33.602787 30148 log.cc:824] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:33.605810 30148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c: No bootstrap required, opened a new log
I20260808 15:42:33.613590 30148 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } }
I20260808 15:42:33.613932 30148 raft_consensus.cc:374] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:33.614059 30148 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96d03f197734d27a7955a93766fad5c, State: Initialized, Role: FOLLOWER
I20260808 15:42:33.614657 30148 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } }
I20260808 15:42:33.614894 30148 raft_consensus.cc:388] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:33.615054 30148 raft_consensus.cc:482] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:33.615216 30148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:33.617359 30148 raft_consensus.cc:504] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } }
I20260808 15:42:33.617841 30148 leader_election.cc:302] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96d03f197734d27a7955a93766fad5c; no voters:
I20260808 15:42:33.618568 30148 leader_election.cc:288] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:33.618731 30153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:33.619581 30153 raft_consensus.cc:686] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [term 1 LEADER]: Becoming Leader. State: Replica: f96d03f197734d27a7955a93766fad5c, State: Running, Role: LEADER
I20260808 15:42:33.620080 30153 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } }
I20260808 15:42:33.620728 30148 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:33.622860 30154 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96d03f197734d27a7955a93766fad5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } } }
I20260808 15:42:33.622927 30155 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96d03f197734d27a7955a93766fad5c. Latest consensus state: current_term: 1 leader_uuid: "f96d03f197734d27a7955a93766fad5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96d03f197734d27a7955a93766fad5c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43837 } } }
I20260808 15:42:33.623293 30155 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:33.623289 30154 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:33.625046 30161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:33.631289 30161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:33.639576 30161 catalog_manager.cc:1347] Generated new cluster ID: 737f9de7c068427f8119224c1b998607
I20260808 15:42:33.639688 30161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:33.664621 30161 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:33.665913 30161 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:33.688676 30161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f96d03f197734d27a7955a93766fad5c: Generated new TSK 0
I20260808 15:42:33.689788 30161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:42:33.722100 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:33.713084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.23.130.62:43837)
0808 15:42:33.713287 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:33.713299 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:33.713792 (+ 493us) server_negotiation.cc:407] Connection header received
0808 15:42:33.714053 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:33.714059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:33.714137 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:33.714301 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:42:33.714843 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.715713 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.717720 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.718225 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.718335 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:33.719082 (+ 747us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:42:33.721395 (+ 2313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37157/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:42:33.721771 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37157/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 15:42:33.722435 30180 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:33.712809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43837 (local address 127.0.0.1:60500)
0808 15:42:33.713651 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:33.713671 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:33.713831 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:33.714335 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:33.714341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:33.714355 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:42:33.714680 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:33.714692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.715875 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.715881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:33.717561 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:33.717577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:33.718460 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:33.718467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:33.718674 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:33.718721 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:42:33.721634 (+ 2913us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37157/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:42:33.721978 (+ 344us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.130.62:43837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37157/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":537,"threads_started":1}
W20260808 15:42:33.722718 30151 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37157/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260808 15:42:33.723239 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30078
I20260808 15:42:33.724380 30143 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 15:42:33.734495 30181 webserver.cc:533] Webserver: sslize failed:
E20260808 15:42:33.734587 30181 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:42:33.819239 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30078
2026-08-08T15:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T15:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:33Z Disabled control of system clock
I20260808 15:42:33.911067 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45815
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:45815
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/testchainca.cert with env {}
W20260808 15:42:34.244396 30187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:34.244699 30187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:34.244851 30187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:34.252261 30187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:34.252396 30187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:34.252487 30187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:34.252594 30187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:34.252686 30187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:42:34.252777 30187 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:42:34.252848 30187 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 15:42:34.262732 30187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:45815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:34.264171 30187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:34.265515 30187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:34.277549 30197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:34.277607 30193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:34.277577 30194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:34.278702 30187 server_base.cc:1034] running on GCE node
I20260808 15:42:34.279469 30187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:34.280819 30187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:34.282063 30187 hybrid_clock.cc:648] HybridClock initialized: now 1786203754282015 us; error 58 us; skew 500 ppm
I20260808 15:42:34.282490 30187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:42:34.284834 30187 webserver.cc:460] Webserver started at http://127.23.130.62:37419/ using document root <none> and password file <none>
I20260808 15:42:34.285544 30187 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:34.285697 30187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:34.286023 30187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:34.288911 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "8e166ab04d32468bb3d5d936f458b178"
format_stamp: "Formatted at 2026-08-08 15:42:34 on dist-test-slave-gp6n"
server_key: "ffea67d9f626bdcb6ca44cce6b736aa1"
server_key_iv: "77b0ff91e15b0d96c02555ead2af9865"
server_key_version: "encryptionkey@0"
I20260808 15:42:34.289742 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "8e166ab04d32468bb3d5d936f458b178"
format_stamp: "Formatted at 2026-08-08 15:42:34 on dist-test-slave-gp6n"
server_key: "ffea67d9f626bdcb6ca44cce6b736aa1"
server_key_iv: "77b0ff91e15b0d96c02555ead2af9865"
server_key_version: "encryptionkey@0"
I20260808 15:42:34.294759 30187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:42:34.297942 30203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:34.299427 30187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:34.299623 30187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "8e166ab04d32468bb3d5d936f458b178"
format_stamp: "Formatted at 2026-08-08 15:42:34 on dist-test-slave-gp6n"
server_key: "ffea67d9f626bdcb6ca44cce6b736aa1"
server_key_iv: "77b0ff91e15b0d96c02555ead2af9865"
server_key_version: "encryptionkey@0"
I20260808 15:42:34.299839 30187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:34.325749 30187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:34.326694 30187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:34.327010 30187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:34.346056 30187 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:45815
I20260808 15:42:34.346071 30258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:45815 every 8 connection(s)
I20260808 15:42:34.347508 30187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:34.350956 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30187
I20260808 15:42:34.351178 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:34.351405 30260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:34.351873 24072 external_mini_cluster.cc:1428] Setting key d5c04df3dc0c97e1468e66e44159408b
I20260808 15:42:34.367466 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:34.353756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.23.130.62:45815)
0808 15:42:34.354938 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:34.354954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:34.355004 (+ 50us) server_negotiation.cc:407] Connection header received
0808 15:42:34.356641 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:34.356670 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:34.357287 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:34.357764 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:34.358372 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.360522 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.361844 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.362975 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.363198 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:34.364421 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:34.364431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:34.364450 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:42:34.365284 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:34.365397 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:42:34.365423 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:34.365524 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:34.365745 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 15:42:34.366095 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":345,"threads_started":1}
I20260808 15:42:34.367576 30260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178: Bootstrap starting.
I20260808 15:42:34.371029 30260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:34.371438 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:42:34.372213 30260 log.cc:824] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:34.375458 30260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178: No bootstrap required, opened a new log
I20260808 15:42:34.382938 30260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } }
I20260808 15:42:34.383328 30260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:34.383446 30260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e166ab04d32468bb3d5d936f458b178, State: Initialized, Role: FOLLOWER
I20260808 15:42:34.384101 30260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } }
I20260808 15:42:34.384392 30260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:34.384554 30260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:34.384723 30260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:34.386962 30260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } }
I20260808 15:42:34.387470 30260 leader_election.cc:302] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e166ab04d32468bb3d5d936f458b178; no voters:
I20260808 15:42:34.388082 30260 leader_election.cc:288] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:34.388242 30266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:34.388954 30266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [term 1 LEADER]: Becoming Leader. State: Replica: 8e166ab04d32468bb3d5d936f458b178, State: Running, Role: LEADER
I20260808 15:42:34.389468 30266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } }
I20260808 15:42:34.390211 30260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:34.393518 30267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e166ab04d32468bb3d5d936f458b178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } } }
I20260808 15:42:34.394268 30267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:34.393853 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e166ab04d32468bb3d5d936f458b178. Latest consensus state: current_term: 1 leader_uuid: "8e166ab04d32468bb3d5d936f458b178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e166ab04d32468bb3d5d936f458b178" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45815 } } }
I20260808 15:42:34.394449 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:34.398032 30279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:34.402541 30279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:34.411095 30279 catalog_manager.cc:1347] Generated new cluster ID: 0caf584c309e428392c37acb83427f95
I20260808 15:42:34.411199 30279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:34.427603 30279 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:34.429301 30279 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:34.437265 30279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e166ab04d32468bb3d5d936f458b178: Generated new TSK 0
I20260808 15:42:34.438084 30279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:42:35.327483 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:34.451794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.23.130.62:45815)
0808 15:42:34.451966 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:34.451971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:34.452362 (+ 391us) server_negotiation.cc:407] Connection header received
0808 15:42:34.452580 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:34.452587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:34.452688 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:34.452873 (+ 185us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:42:34.453287 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.454633 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.456073 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.456883 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.457027 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:34.457495 (+ 468us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:42:35.326872 (+869377us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:42:35.327235 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 15:42:35.327647 30263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 15:42:35.327898 30294 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:34.451633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45815 (local address 127.0.0.1:53560)
0808 15:42:34.452249 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:34.452265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:34.452387 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:34.452885 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:34.452888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:34.452898 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:42:34.453138 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:34.453145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.454800 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.454804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:34.455916 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:34.455927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:34.457049 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:34.457053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:34.457170 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:34.457200 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:42:35.327187 (+869987us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:42:35.327444 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.130.62:45815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":331,"threads_started":1}
I20260808 15:42:35.329257 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30187
I20260808 15:42:35.330466 30254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:35.433094 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30187
2026-08-08T15:42:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1595 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (418 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (435 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:36 dist-test-slave-gp6n krb5kdc[30309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:36 dist-test-slave-gp6n krb5kdc[30309](info): set up 2 sockets
Aug 08 15:42:36 dist-test-slave-gp6n krb5kdc[30309](info): commencing operation
krb5kdc: starting...
W20260808 15:42:37.445261 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:37 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:37.677598 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44371
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:44371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:38.013775 30322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:38.014078 30322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:38.014169 30322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:38.022255 30322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:38.022390 30322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:38.022517 30322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:38.022588 30322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:38.022632 30322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:38.033236 30322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:44371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:38.034956 30322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:38.036334 30322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:38.048801 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:38.048887 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:38.048842 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:38.055131 30322 server_base.cc:1034] running on GCE node
I20260808 15:42:38.055930 30322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:38.057397 30322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:38.058599 30322 hybrid_clock.cc:648] HybridClock initialized: now 1786203758058548 us; error 37 us; skew 500 ppm
I20260808 15:42:38.058964 30322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:38 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203758, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:38.066243 30322 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:42:38.067862 30322 webserver.cc:460] Webserver started at http://127.23.130.62:34901/ using document root <none> and password file <none>
I20260808 15:42:38.068595 30322 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:38.068746 30322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:38.069077 30322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:38.072208 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "603756ae663448c4b72252a1771bed0a"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "cba2cedf73eaf1d759f42300d32220bb"
server_key_iv: "5801c893e6f98a9989aa3ecd54a41a8f"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.073016 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "603756ae663448c4b72252a1771bed0a"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "cba2cedf73eaf1d759f42300d32220bb"
server_key_iv: "5801c893e6f98a9989aa3ecd54a41a8f"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.078168 30322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:42:38.081724 30338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:38.083029 30322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:38.083211 30322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "603756ae663448c4b72252a1771bed0a"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "cba2cedf73eaf1d759f42300d32220bb"
server_key_iv: "5801c893e6f98a9989aa3ecd54a41a8f"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.083412 30322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:38.110944 30322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:38.115409 30322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:38.115715 30322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:38.135062 30322 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:44371
I20260808 15:42:38.135108 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:44371 every 8 connection(s)
I20260808 15:42:38.136461 30322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:38.140273 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:38.141355 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30322
I20260808 15:42:38.141600 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:38.142498 24072 external_mini_cluster.cc:1428] Setting key e188e4f559c0dbfd73de092af9080a91
I20260808 15:42:38.155400 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a: Bootstrap starting.
Aug 08 15:42:38 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:38.159003 30396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:38.160238 30396 log.cc:824] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:38.163121 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a: No bootstrap required, opened a new log
I20260808 15:42:38.165695 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:38.144589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.23.130.62:44371)
0808 15:42:38.145407 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:38.145419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:38.145464 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:42:38.147131 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:38.147156 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:38.147547 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:38.147868 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:38.149648 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.151004 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.152803 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.153485 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.153611 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:38.158342 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:38.158349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:38.158364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:38.158445 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:38.161575 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:38.162258 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:38.162270 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:38.162276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:38.162505 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:38.162892 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:38.162896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:38.162900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:38.163437 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:38.163790 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:38.164247 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 15:42:38.164688 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260808 15:42:38.170954 30396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } }
I20260808 15:42:38.171303 30396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:38.171419 30396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603756ae663448c4b72252a1771bed0a, State: Initialized, Role: FOLLOWER
I20260808 15:42:38.172015 30396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } }
I20260808 15:42:38.172256 30396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:38.172423 30396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:38.172626 30396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:38.174782 30396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } }
I20260808 15:42:38.175249 30396 leader_election.cc:302] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603756ae663448c4b72252a1771bed0a; no voters:
I20260808 15:42:38.175828 30396 leader_election.cc:288] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:38.175937 30401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:38.177289 30401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [term 1 LEADER]: Becoming Leader. State: Replica: 603756ae663448c4b72252a1771bed0a, State: Running, Role: LEADER
I20260808 15:42:38.177783 30401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } }
I20260808 15:42:38.177922 30396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:38.180601 30403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603756ae663448c4b72252a1771bed0a. Latest consensus state: current_term: 1 leader_uuid: "603756ae663448c4b72252a1771bed0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } } }
I20260808 15:42:38.181231 30403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:38.181034 30402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603756ae663448c4b72252a1771bed0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603756ae663448c4b72252a1771bed0a" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 44371 } } }
I20260808 15:42:38.181331 30402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:38.182020 30411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:38.187815 30411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:38.195593 30411 catalog_manager.cc:1347] Generated new cluster ID: 00a7fd8150ec41a8a8d7b6d485170370
I20260808 15:42:38.195705 30411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:38.230265 30411 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:38.231638 30411 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:38.245338 30411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603756ae663448c4b72252a1771bed0a: Generated new TSK 0
I20260808 15:42:38.246186 30411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:42:38.309573 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:38.645694 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:38.645954 30423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:38.646049 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:38.653278 30423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:38.653389 30423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:38.653510 30423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:38.663280 30423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:38.664536 30423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:38.666205 30423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:38.679908 30429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:38.679996 30432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:38.680037 30430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:38.680943 30423 server_base.cc:1034] running on GCE node
I20260808 15:42:38.681622 30423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:38.682637 30423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:38.683836 30423 hybrid_clock.cc:648] HybridClock initialized: now 1786203758683788 us; error 37 us; skew 500 ppm
I20260808 15:42:38.684106 30423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:38 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203758, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:38.687305 30423 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:42:38.688870 30423 webserver.cc:460] Webserver started at http://127.23.130.1:34249/ using document root <none> and password file <none>
I20260808 15:42:38.689524 30423 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:38.689630 30423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:38.689880 30423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:38.692809 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "fb90aff33923477a9ad1983799c86754"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "f4509b535de4094b9dbeaf17e03488a0"
server_key_iv: "8b926294bc231513777f074f0cb81be1"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.693538 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "fb90aff33923477a9ad1983799c86754"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "f4509b535de4094b9dbeaf17e03488a0"
server_key_iv: "8b926294bc231513777f074f0cb81be1"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.698462 30423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:42:38.701756 30440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:38.703140 30423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:38.703267 30423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "fb90aff33923477a9ad1983799c86754"
format_stamp: "Formatted at 2026-08-08 15:42:38 on dist-test-slave-gp6n"
server_key: "f4509b535de4094b9dbeaf17e03488a0"
server_key_iv: "8b926294bc231513777f074f0cb81be1"
server_key_version: "encryptionkey@0"
I20260808 15:42:38.703428 30423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:38.716874 30423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:38.720634 30423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:38.720870 30423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:38.721707 30423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:38.723105 30423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:38.723178 30423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:38.723260 30423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:38.723316 30423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:38.757475 30423 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:34887
I20260808 15:42:38.757565 30559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:34887 every 8 connection(s)
I20260808 15:42:38.758865 30423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:38.763091 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30423
I20260808 15:42:38.763334 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:38.763993 24072 external_mini_cluster.cc:1428] Setting key de7ab17977ce2361b794853dca1ea28a
Aug 08 15:42:38 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203758, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:38.781038 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:38.761425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:46173 (local address 127.23.130.62:44371)
0808 15:42:38.761865 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:38.761870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:38.762779 (+ 909us) server_negotiation.cc:407] Connection header received
0808 15:42:38.764794 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:38.764800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:38.764890 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:38.765009 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:38.766935 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.768166 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.769774 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.770509 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.770653 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:38.774228 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:38.774236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:38.774244 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:38.774289 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:38.777463 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:38.778570 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:38.778574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:38.778576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:38.778696 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:38.779261 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:38.779264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:38.779267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:38.779529 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:38.779644 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:38.780612 (+ 968us) server_negotiation.cc:299] Negotiation successful
0808 15:42:38.780780 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
I20260808 15:42:38.782902 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:38.761860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44371 (local address 127.23.130.1:46173)
0808 15:42:38.762623 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:38.762669 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:38.764453 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:38.765303 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:38.765314 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:38.766077 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:38.766761 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:38.766780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.768318 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.768321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:38.769624 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:38.769636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:38.770706 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:38.770711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:38.770909 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:38.771623 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:38.771645 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:38.774002 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:38.777629 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:38.777642 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:38.777662 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:38.778445 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:38.778847 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:38.778855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:38.778862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:38.779138 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:38.779675 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:38.779686 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:38.780187 (+ 501us) client_negotiation.cc:769] Sending connection context
0808 15:42:38.780520 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 15:42:38.781184 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":289,"threads_started":1}
I20260808 15:42:38.784600 30560 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44371
I20260808 15:42:38.784919 30560 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:38.785665 30560 heartbeater.cc:511] Master 127.23.130.62:44371 requested a full tablet report, sending...
I20260808 15:42:38.788206 30356 ts_manager.cc:194] Registered new tserver with Master: fb90aff33923477a9ad1983799c86754 (127.23.130.1:34887)
I20260808 15:42:38.789661 30356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:46173
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:42:38.826289 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:39.159600 30567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:39.159921 30567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:39.160097 30567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:39.167377 30567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:39.167539 30567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:39.167712 30567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:39.177377 30567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:39.178815 30567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:39.180184 30567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:39.193984 30572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:39.193907 30573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:39.193885 30575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:39.194687 30567 server_base.cc:1034] running on GCE node
I20260808 15:42:39.195277 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:39.196216 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:39.197444 30567 hybrid_clock.cc:648] HybridClock initialized: now 1786203759197388 us; error 52 us; skew 500 ppm
I20260808 15:42:39.197757 30567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.200944 30567 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:42:39.202589 30567 webserver.cc:460] Webserver started at http://127.23.130.2:33631/ using document root <none> and password file <none>
I20260808 15:42:39.203271 30567 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:39.203421 30567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:39.203737 30567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:39.206692 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "7494a61eb06e4417981dd154e06d5558"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "b149461f1bf8a67ad14fcf2129f15e9a"
server_key_iv: "f769caefa81e22f69dc4084bb5557eda"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.207506 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "7494a61eb06e4417981dd154e06d5558"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "b149461f1bf8a67ad14fcf2129f15e9a"
server_key_iv: "f769caefa81e22f69dc4084bb5557eda"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.212787 30567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:42:39.216467 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.217796 30567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:39.217980 30567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "7494a61eb06e4417981dd154e06d5558"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "b149461f1bf8a67ad14fcf2129f15e9a"
server_key_iv: "f769caefa81e22f69dc4084bb5557eda"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.218182 30567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:39.239527 30567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:39.243801 30567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:39.244119 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:39.245157 30567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:39.246739 30567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:39.246843 30567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.246997 30567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:39.247071 30567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.281904 30567 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:46825
I20260808 15:42:39.281960 30704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:46825 every 8 connection(s)
I20260808 15:42:39.283341 30567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:39.292045 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30567
I20260808 15:42:39.292269 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:39.292747 24072 external_mini_cluster.cc:1428] Setting key 9b636c3531d28c50fb65e50b03db74b0
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:39.304464 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.286275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:60029 (local address 127.23.130.62:44371)
0808 15:42:39.286754 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.286763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.287996 (+ 1233us) server_negotiation.cc:407] Connection header received
0808 15:42:39.289849 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.289854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.289944 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.290074 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.291620 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.292478 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.294206 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.294796 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.294930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.298712 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.298717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.298720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.298750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.301369 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.302115 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.302120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.302122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.302249 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.302736 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.302739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.302741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.303008 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.303172 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.303843 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.304072 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260808 15:42:39.305413 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.286962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44371 (local address 127.23.130.2:60029)
0808 15:42:39.287788 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:39.287869 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:39.289578 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:39.290321 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:39.290329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:39.290893 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:39.291437 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:39.291466 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.292631 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.292636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:39.294055 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:39.294067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.295020 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.295028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:39.295264 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.295955 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:39.295982 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:39.298485 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:39.301522 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:39.301531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:39.301546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:39.301977 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:39.302364 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:39.302369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:39.302373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:39.302604 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:39.303142 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:39.303150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:39.303479 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 15:42:39.303746 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 15:42:39.304278 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
I20260808 15:42:39.307070 30706 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44371
I20260808 15:42:39.307421 30706 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:39.308203 30706 heartbeater.cc:511] Master 127.23.130.62:44371 requested a full tablet report, sending...
I20260808 15:42:39.309755 30356 ts_manager.cc:194] Registered new tserver with Master: 7494a61eb06e4417981dd154e06d5558 (127.23.130.2:46825)
I20260808 15:42:39.311218 30356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:60029
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:42:39.349179 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:39.683688 30713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:39.684005 30713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:39.684177 30713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:39.691735 30713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:39.691877 30713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:39.692046 30713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:39.701556 30713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:39.702960 30713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:39.704291 30713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:39.717994 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:39.718021 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:39.718042 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:39.718840 30713 server_base.cc:1034] running on GCE node
I20260808 15:42:39.719483 30713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:39.720487 30713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:39.721712 30713 hybrid_clock.cc:648] HybridClock initialized: now 1786203759721652 us; error 50 us; skew 500 ppm
I20260808 15:42:39.722081 30713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.725402 30713 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:42:39.727030 30713 webserver.cc:460] Webserver started at http://127.23.130.3:42539/ using document root <none> and password file <none>
I20260808 15:42:39.727702 30713 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:39.727870 30713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:39.728181 30713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:39.731230 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "092aad03c1c043f5a64552187f0e0d1a"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "172b008a52ee1aac0f9d143b3d011bf9"
server_key_iv: "dbf6efd8eb6b3d66e5008052c4b28418"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.732025 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "092aad03c1c043f5a64552187f0e0d1a"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "172b008a52ee1aac0f9d143b3d011bf9"
server_key_iv: "dbf6efd8eb6b3d66e5008052c4b28418"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.737222 30713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:42:39.741024 30730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.742344 30713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:42:39.742547 30713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "092aad03c1c043f5a64552187f0e0d1a"
format_stamp: "Formatted at 2026-08-08 15:42:39 on dist-test-slave-gp6n"
server_key: "172b008a52ee1aac0f9d143b3d011bf9"
server_key_iv: "dbf6efd8eb6b3d66e5008052c4b28418"
server_key_version: "encryptionkey@0"
I20260808 15:42:39.742758 30713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:39.765700 30713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:39.770157 30713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:39.770562 30713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:39.771603 30713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:39.772936 30713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:39.773051 30713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.773205 30713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:39.773278 30713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:39.792656 30560 heartbeater.cc:503] Master 127.23.130.62:44371 was elected leader, sending a full tablet report...
I20260808 15:42:39.812292 30713 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:32953
I20260808 15:42:39.812376 30854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:32953 every 8 connection(s)
I20260808 15:42:39.813580 30713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:39.821728 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30713
I20260808 15:42:39.821943 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:39.822537 24072 external_mini_cluster.cc:1428] Setting key 3d012aa078c4308625b73e11172b31d3
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:42:39.833985 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.816387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:37949 (local address 127.23.130.62:44371)
0808 15:42:39.816995 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.817003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.818011 (+ 1008us) server_negotiation.cc:407] Connection header received
0808 15:42:39.819908 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.819914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.820015 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.820148 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.821912 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.823019 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.824792 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.825210 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.825348 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.828868 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.828873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.828876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.828913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.831372 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.832064 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.832068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.832070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.832181 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.832550 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.832553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.832555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.832796 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.832915 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.833540 (+ 625us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.833700 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":274,"threads_started":1}
I20260808 15:42:39.834995 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.817079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:44371 (local address 127.23.130.3:37949)
0808 15:42:39.817844 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:39.817883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:39.819640 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:39.820384 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:39.820392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:39.820907 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:39.821670 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:39.821700 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.823187 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.823191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:39.824631 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:39.824644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.825441 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.825450 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:39.825688 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.826374 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:39.826397 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:39.828680 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:39.831548 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:39.831554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:39.831566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:39.831966 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:39.832270 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:39.832273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:39.832275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:39.832440 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:39.832907 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:39.832912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:39.833198 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 15:42:39.833428 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 15:42:39.833819 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":283,"threads_started":1}
I20260808 15:42:39.836390 30855 heartbeater.cc:347] Connected to a master server at 127.23.130.62:44371
I20260808 15:42:39.836712 30855 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:39.837476 30855 heartbeater.cc:511] Master 127.23.130.62:44371 requested a full tablet report, sending...
I20260808 15:42:39.839874 30356 ts_manager.cc:194] Registered new tserver with Master: 092aad03c1c043f5a64552187f0e0d1a (127.23.130.3:32953)
I20260808 15:42:39.840674 30356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:37949
I20260808 15:42:39.850060 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/test-admin.keytab.
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.881875 24072 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 15:42:39.887607 30864 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:39.884854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.23.130.1:34887)
0808 15:42:39.885559 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.885567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.885591 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:42:39.885657 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.885664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.885874 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.886050 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.887045 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33164: BlockingRecv error: recv got EOF from 127.0.0.1:33164 (error 108)
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":292,"threads_started":1}
W20260808 15:42:39.888029 24072 init.cc:244] Error while opening credential cache '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc)
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.889788 24072 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.891206 24072 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:39.905794 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.892820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.23.130.1:34887)
0808 15:42:39.893015 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.893019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.893150 (+ 131us) server_negotiation.cc:407] Connection header received
0808 15:42:39.893273 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.893276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.893332 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.893416 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.894333 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.895341 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.896530 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.897076 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.897170 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.900585 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.900596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.900609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.900647 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.903399 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.904001 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.904008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.904010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.904130 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.904442 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.904446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.904462 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.904792 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.905005 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.905254 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.905524 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 15:42:39.911613 30864 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:39.910020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.23.130.1:34887)
0808 15:42:39.910217 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.910224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.910437 (+ 213us) server_negotiation.cc:407] Connection header received
0808 15:42:39.910619 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.910623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.910686 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.910762 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.911387 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 15:42:39.911757 24072 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.913273 24072 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.914662 24072 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:39.929468 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.915873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.23.130.1:34887)
0808 15:42:39.916107 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.916114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.916382 (+ 268us) server_negotiation.cc:407] Connection header received
0808 15:42:39.916561 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.916567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.916638 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.916752 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.917765 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.918558 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.919733 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.920181 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.920334 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.924095 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.924102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.924106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.924150 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.926922 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.927587 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.927594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.927598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.927727 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.928118 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.928125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.928128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.928409 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.928669 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.928963 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.929165 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260808 15:42:39.933531 30864 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:39.931959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.23.130.1:34887)
0808 15:42:39.932176 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.932182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.932390 (+ 208us) server_negotiation.cc:407] Connection header received
0808 15:42:39.932509 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.932512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.932577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.932653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.933306 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33208: BlockingRecv error: recv got EOF from 127.0.0.1:33208 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260808 15:42:39.933706 24072 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786203706771851-24072-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.935158 24072 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:39.936394 24072 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:42:39 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:42:39.949560 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.937581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.23.130.1:34887)
0808 15:42:39.937796 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.937801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.938023 (+ 222us) server_negotiation.cc:407] Connection header received
0808 15:42:39.938131 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.938134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.938184 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.938258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.939166 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.939859 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.941043 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.941437 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.941527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.944733 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.944738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.944741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.944771 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.947327 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.947933 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.947937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.947939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.948074 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.948371 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.948374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.948376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.948642 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.948863 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.949047 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.949224 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:42:39.951198 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30423
I20260808 15:42:39.962612 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:39.951882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.23.130.1:34887)
0808 15:42:39.952020 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:39.952024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:39.952435 (+ 411us) server_negotiation.cc:407] Connection header received
0808 15:42:39.952556 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:39.952560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:39.952609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:39.952684 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:39.953560 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.954251 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.955453 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:39.955834 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:39.955944 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:39.957768 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:39.957772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:39.957775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:39.957802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:39.960210 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.960833 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.960838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.960840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.960985 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:39.961372 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:39.961376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:39.961378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:39.961653 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:39.961864 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:39.962090 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 15:42:39.962250 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:42:39.963151 30555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:40.047590 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30423
I20260808 15:42:40.066593 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30567
Aug 08 15:42:40 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:42:40.083640 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:40.067580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.23.130.2:46825)
0808 15:42:40.068180 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:40.068187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:40.068214 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:42:40.068275 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:40.068282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:40.068501 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:40.068629 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:40.069710 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:40.071020 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:40.072449 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:40.073009 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:40.073109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:40.077395 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:40.077406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:40.077420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:40.077474 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:40.080454 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:40.081131 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:40.081142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:40.081144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:40.081279 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:40.081689 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:40.081693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:40.081695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:40.082054 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:40.082344 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:40.082929 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 15:42:40.083223 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":337,"threads_started":1}
I20260808 15:42:40.084995 30700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:40.169859 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30567
I20260808 15:42:40.188848 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30713
Aug 08 15:42:40 dist-test-slave-gp6n krb5kdc[30309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:42:40.205561 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:40.190045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.23.130.3:32953)
0808 15:42:40.190739 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:40.190746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:40.190776 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:42:40.190836 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:40.190843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:40.191071 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:40.191243 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:40.192387 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:40.193413 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:40.194933 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:40.195508 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:40.195650 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:40.199686 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:40.199696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:40.199710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:40.199748 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:40.202791 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:40.203498 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:40.203514 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:40.203516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:40.203639 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:40.204043 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:40.204051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:40.204054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:40.204391 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:40.204728 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:40.204997 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 15:42:40.205262 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":394,"threads_started":1}
I20260808 15:42:40.206948 30850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:40.302616 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30713
I20260808 15:42:40.323553 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30322
I20260808 15:42:40.324851 30391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:40.430064 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30322
2026-08-08T15:42:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4147 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:40 dist-test-slave-gp6n krb5kdc[30887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:40 dist-test-slave-gp6n krb5kdc[30887](info): set up 2 sockets
Aug 08 15:42:40 dist-test-slave-gp6n krb5kdc[30887](info): commencing operation
krb5kdc: starting...
W20260808 15:42:41.588876 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:41 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab.
Entry for principal oryx/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab.
I20260808 15:42:41.829887 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:46165
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab
--principal=oryx/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:46165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:42.159490 30902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:42.159770 30902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:42.159947 30902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:42.167052 30902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:42.167160 30902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:42.167263 30902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:42.167356 30902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:42.167448 30902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:42.177230 30902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:46165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab
--principal=oryx/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:42.178562 30902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:42.179937 30902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:42.192003 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:42.192063 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:42.192041 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:42.193055 30902 server_base.cc:1034] running on GCE node
I20260808 15:42:42.193845 30902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:42.195289 30902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:42.196494 30902 hybrid_clock.cc:648] HybridClock initialized: now 1786203762196439 us; error 50 us; skew 500 ppm
I20260808 15:42:42.196839 30902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:42 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203762, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:42.200148 30902 init.cc:377] Logged in from keytab as oryx/127.23.130.62@KRBTEST.COM (short username oryx)
I20260808 15:42:42.201592 30902 webserver.cc:460] Webserver started at http://127.23.130.62:43511/ using document root <none> and password file <none>
I20260808 15:42:42.202231 30902 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:42.202332 30902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:42.202656 30902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:42.205497 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "c35d5c8533344b349a964b9002cf0401"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "ca3227a79be8f08e6802b041238d1dd5"
server_key_iv: "8aed6fe7aaad069c55c141dd9041cdf6"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.206202 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "c35d5c8533344b349a964b9002cf0401"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "ca3227a79be8f08e6802b041238d1dd5"
server_key_iv: "8aed6fe7aaad069c55c141dd9041cdf6"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.211356 30902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:42:42.214783 30919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:42.216084 30902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:42:42.216295 30902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "c35d5c8533344b349a964b9002cf0401"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "ca3227a79be8f08e6802b041238d1dd5"
server_key_iv: "8aed6fe7aaad069c55c141dd9041cdf6"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.216576 30902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:42.234025 30902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:42.238628 30902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:42.238989 30902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:42.258068 30902 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:46165
I20260808 15:42:42.258101 30971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:46165 every 8 connection(s)
I20260808 15:42:42.259909 30902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:42.261878 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 30902
I20260808 15:42:42.262133 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:42.263108 24072 external_mini_cluster.cc:1428] Setting key e0180d8db1c2daa442289a6b09a737ff
I20260808 15:42:42.264747 30972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:42.276844 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Bootstrap starting.
Aug 08 15:42:42 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:42.280045 30972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:42.281037 30972 log.cc:824] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:42.283804 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: No bootstrap required, opened a new log
I20260808 15:42:42.285357 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:42.265529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.23.130.62:46165)
0808 15:42:42.266349 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:42.266361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:42.266463 (+ 102us) server_negotiation.cc:407] Connection header received
0808 15:42:42.268082 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:42.268105 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:42.268488 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:42.268891 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:42.270440 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.271758 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.272976 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.273630 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.273841 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:42.277995 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:42.278003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:42.278017 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:42.278053 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:42.281305 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:42.282027 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:42.282042 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:42.282047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:42.282208 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:42.282672 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:42.282677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:42.282682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:42.283197 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:42.283532 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:42.283984 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 15:42:42.284409 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":485,"threads_started":1}
I20260808 15:42:42.292641 30972 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } }
I20260808 15:42:42.293056 30972 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:42.293164 30972 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d5c8533344b349a964b9002cf0401, State: Initialized, Role: FOLLOWER
I20260808 15:42:42.293720 30972 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } }
I20260808 15:42:42.293992 30972 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:42.294133 30972 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:42.294351 30972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:42.296746 30972 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } }
I20260808 15:42:42.297220 30972 leader_election.cc:302] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35d5c8533344b349a964b9002cf0401; no voters:
I20260808 15:42:42.297812 30972 leader_election.cc:288] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:42.297966 30977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:42.298724 30977 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 LEADER]: Becoming Leader. State: Replica: c35d5c8533344b349a964b9002cf0401, State: Running, Role: LEADER
I20260808 15:42:42.299305 30977 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } }
I20260808 15:42:42.300041 30972 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:42.302137 30978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } }
I20260808 15:42:42.302127 30979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35d5c8533344b349a964b9002cf0401. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } }
I20260808 15:42:42.302546 30979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:42.302546 30978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:42.303458 30982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:42.309966 30982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:42.319077 30982 catalog_manager.cc:1347] Generated new cluster ID: 68cf3bdc3afd467b817b4137a427bcf7
I20260808 15:42:42.319223 30982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:42.330302 30982 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:42.331616 30982 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:42.338128 30982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Generated new TSK 0
I20260808 15:42:42.338990 30982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
I20260808 15:42:42.406810 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:42.743955 31001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:42.744240 31001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:42.744335 31001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:42.751792 31001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:42.751904 31001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:42.752017 31001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:42.761722 31001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:42.763032 31001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:42.764317 31001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:42.777705 31009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:42.777750 31006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:42.777779 31007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:42.778633 31001 server_base.cc:1034] running on GCE node
I20260808 15:42:42.779177 31001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:42.780082 31001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:42.781302 31001 hybrid_clock.cc:648] HybridClock initialized: now 1786203762781230 us; error 55 us; skew 500 ppm
I20260808 15:42:42.781709 31001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:42 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203762, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:42.785154 31001 init.cc:377] Logged in from keytab as oryx/127.23.130.1@KRBTEST.COM (short username oryx)
I20260808 15:42:42.786649 31001 webserver.cc:460] Webserver started at http://127.23.130.1:33725/ using document root <none> and password file <none>
I20260808 15:42:42.787436 31001 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:42.787544 31001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:42.787788 31001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:42.790753 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "1681ff6962684d8db69f0265bdd39310"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "8e1745f45bef3fd5cdf2c796c479e527"
server_key_iv: "ba68b1be87cc9ea4f9118f9a80a3ac42"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.791576 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "1681ff6962684d8db69f0265bdd39310"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "8e1745f45bef3fd5cdf2c796c479e527"
server_key_iv: "ba68b1be87cc9ea4f9118f9a80a3ac42"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.796793 31001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:42:42.800302 31016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:42.801890 31001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:42:42.802032 31001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "1681ff6962684d8db69f0265bdd39310"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "8e1745f45bef3fd5cdf2c796c479e527"
server_key_iv: "ba68b1be87cc9ea4f9118f9a80a3ac42"
server_key_version: "encryptionkey@0"
I20260808 15:42:42.802191 31001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:42.833665 31001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:42.838191 31001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:42.838563 31001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:42.839618 31001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:42.840943 31001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:42.841029 31001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:42.841164 31001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:42.841235 31001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:42.875649 31001 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:40311
I20260808 15:42:42.875749 31143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:40311 every 8 connection(s)
I20260808 15:42:42.876991 31001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:42.883287 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31001
I20260808 15:42:42.883572 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:42.884318 24072 external_mini_cluster.cc:1428] Setting key a43d6fde71c515ffe7d8edbcee53cf0d
Aug 08 15:42:42 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203762, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:42.897368 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:42.879633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:51059 (local address 127.23.130.62:46165)
0808 15:42:42.880121 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:42.880126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:42.880943 (+ 817us) server_negotiation.cc:407] Connection header received
0808 15:42:42.882918 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:42.882926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:42.883040 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:42.883192 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:42.884852 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.885849 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.887557 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.887963 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.888103 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:42.891583 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:42.891588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:42.891592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:42.891623 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:42.894202 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:42.894977 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:42.894981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:42.894983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:42.895102 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:42.895615 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:42.895620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:42.895622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:42.895884 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:42.896054 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:42.896747 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 15:42:42.896984 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
I20260808 15:42:42.898334 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:42.880046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:51059)
0808 15:42:42.880742 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:42.880779 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:42.882546 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:42.883409 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:42.883422 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:42.884029 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:42.884662 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:42.884683 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.886012 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.886017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:42.887389 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:42.887401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:42.888202 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:42.888209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:42.888435 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:42.889141 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:42.889164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:42.891351 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:42.894348 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:42.894356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:42.894372 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:42.894830 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:42.895218 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:42.895223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:42.895226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:42.895482 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:42.896025 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:42.896033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:42.896368 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 15:42:42.896638 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 15:42:42.897146 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":276,"threads_started":1}
I20260808 15:42:42.900046 31144 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:42.900362 31144 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:42.901223 31144 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:42.903347 30936 ts_manager.cc:194] Registered new tserver with Master: 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311)
I20260808 15:42:42.904827 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:51059
WARNING: no policy specified for oryx/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
I20260808 15:42:42.939209 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:43.275794 31150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:43.276062 31150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:43.276156 31150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:43.283386 31150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:43.283490 31150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:43.283599 31150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:43.293610 31150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:43.294868 31150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:43.296176 31150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:43.309579 31160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:43.309544 31158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:43.309614 31157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:43.311990 31150 server_base.cc:1034] running on GCE node
I20260808 15:42:43.312619 31150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:43.313512 31150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:43.314755 31150 hybrid_clock.cc:648] HybridClock initialized: now 1786203763314701 us; error 48 us; skew 500 ppm
I20260808 15:42:43.315089 31150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:43 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203763, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:43.323522 31150 init.cc:377] Logged in from keytab as oryx/127.23.130.2@KRBTEST.COM (short username oryx)
I20260808 15:42:43.325148 31150 webserver.cc:460] Webserver started at http://127.23.130.2:38191/ using document root <none> and password file <none>
I20260808 15:42:43.325893 31150 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:43.326043 31150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:43.326351 31150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:43.329348 31150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "24c502541071445aa8c9805607dd731f"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "ae51ce0bebfe33d4ab17c8ec01e21b5d"
server_key_iv: "7e209477b6584d133c3c143dbc7cb266"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.330174 31150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "24c502541071445aa8c9805607dd731f"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "ae51ce0bebfe33d4ab17c8ec01e21b5d"
server_key_iv: "7e209477b6584d133c3c143dbc7cb266"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.335459 31150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:42:43.339112 31167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:43.340521 31150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:43.340732 31150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "24c502541071445aa8c9805607dd731f"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "ae51ce0bebfe33d4ab17c8ec01e21b5d"
server_key_iv: "7e209477b6584d133c3c143dbc7cb266"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.340962 31150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:43.359760 31150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:43.363736 31150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:43.364032 31150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:43.364956 31150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:43.366312 31150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:43.366448 31150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 15:42:43.366603 31150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:43.366678 31150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:43.400794 31150 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:34787
I20260808 15:42:43.400830 31291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:34787 every 8 connection(s)
I20260808 15:42:43.402132 31150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:43.403844 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31150
I20260808 15:42:43.404089 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:43.404562 24072 external_mini_cluster.cc:1428] Setting key 847be421c1d419fe813de2c62bc83177
Aug 08 15:42:43 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203763, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:43.426853 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:43.404948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:54867 (local address 127.23.130.62:46165)
0808 15:42:43.405565 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:43.405569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:43.406816 (+ 1247us) server_negotiation.cc:407] Connection header received
0808 15:42:43.409793 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:43.409797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:43.409870 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:43.409968 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:43.412123 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.412832 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.415374 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.415791 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.415882 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:43.419613 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:43.419623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:43.419631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:43.419676 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:43.423494 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:43.424231 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:43.424239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:43.424245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:43.424420 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:43.424827 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:43.424834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:43.424839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:43.425229 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:43.425400 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:43.426045 (+ 645us) server_negotiation.cc:299] Negotiation successful
0808 15:42:43.426300 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
I20260808 15:42:43.427431 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:43.405676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:54867)
0808 15:42:43.406633 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:43.406686 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:43.409484 (+ 2798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:43.410253 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:43.410267 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:43.411101 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:43.411932 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:43.411973 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.413002 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.413010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:43.415220 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:43.415237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.416024 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.416033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:43.416362 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:43.417138 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:43.417166 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:43.419350 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:43.423629 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:43.423635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:43.423648 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:43.424079 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:43.424529 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:43.424532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:43.424535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:43.424709 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:43.425355 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:43.425361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:43.425667 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 15:42:43.425889 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 15:42:43.426273 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":334,"threads_started":1}
I20260808 15:42:43.428977 31292 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:43.429304 31292 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:43.430055 31292 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:43.431605 30936 ts_manager.cc:194] Registered new tserver with Master: 24c502541071445aa8c9805607dd731f (127.23.130.2:34787)
WARNING: no policy specified for oryx/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:42:43.432498 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:54867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
I20260808 15:42:43.462924 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:43.794759 31298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:43.795037 31298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:43.795199 31298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:43.802470 31298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:43.802573 31298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:43.802740 31298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:43.812446 31298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:43.813719 31298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:43.815069 31298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:43.828630 31304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:43.828734 31306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:43.828652 31303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:43.829252 31298 server_base.cc:1034] running on GCE node
I20260808 15:42:43.829855 31298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:43.830847 31298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:43.832084 31298 hybrid_clock.cc:648] HybridClock initialized: now 1786203763832043 us; error 38 us; skew 500 ppm
I20260808 15:42:43.832402 31298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:43 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203763, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:43.835717 31298 init.cc:377] Logged in from keytab as oryx/127.23.130.3@KRBTEST.COM (short username oryx)
I20260808 15:42:43.837190 31298 webserver.cc:460] Webserver started at http://127.23.130.3:42435/ using document root <none> and password file <none>
I20260808 15:42:43.837885 31298 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:43.838037 31298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:43.838362 31298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:43.841277 31298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "aa0c4cb87e7547c5b08393456f710052"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "c1a0c57f4fa473e8f47f2ec4392dbbd0"
server_key_iv: "742d62c9116fc51c6fe8871777c4c34f"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.842059 31298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "aa0c4cb87e7547c5b08393456f710052"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "c1a0c57f4fa473e8f47f2ec4392dbbd0"
server_key_iv: "742d62c9116fc51c6fe8871777c4c34f"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.847262 31298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:42:43.850713 31314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:43.852025 31298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:42:43.852202 31298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "aa0c4cb87e7547c5b08393456f710052"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "c1a0c57f4fa473e8f47f2ec4392dbbd0"
server_key_iv: "742d62c9116fc51c6fe8871777c4c34f"
server_key_version: "encryptionkey@0"
I20260808 15:42:43.852408 31298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:43.878041 31298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:43.883070 31298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:43.883409 31298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:43.884413 31298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:43.885840 31298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:43.885921 31298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:43.886121 31298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:43.886179 31298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:43.908051 31144 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
I20260808 15:42:43.919663 31298 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:34879
I20260808 15:42:43.919706 31436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:34879 every 8 connection(s)
I20260808 15:42:43.921066 31298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:43.927132 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31298
I20260808 15:42:43.927373 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:43.928040 24072 external_mini_cluster.cc:1428] Setting key eb8aef55658e59c2de5504ee130791fa
Aug 08 15:42:43 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203763, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:43.942384 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:43.924132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:53611 (local address 127.23.130.62:46165)
0808 15:42:43.924311 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:43.924316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:43.925615 (+ 1299us) server_negotiation.cc:407] Connection header received
0808 15:42:43.927561 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:43.927569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:43.927678 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:43.927821 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:43.929656 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.930523 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.932094 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.932508 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.932644 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:43.936419 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:43.936425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:43.936428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:43.936463 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:43.939178 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:43.939974 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:43.939986 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:43.939989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:43.940106 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:43.940562 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:43.940568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:43.940571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:43.940859 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:43.941038 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:43.941780 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 15:42:43.942028 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:42:43.943638 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:43.924711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:53611)
0808 15:42:43.925450 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:43.925489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:43.927218 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:43.928076 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:43.928090 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:43.928677 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:43.929490 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:43.929511 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.930675 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.930687 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:43.931945 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:43.931957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:43.932742 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:43.932749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:43.933000 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:43.933796 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:43.933819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:43.936173 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:43.939327 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:43.939337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:43.939356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:43.939830 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:43.940220 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:43.940225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:43.940228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:43.940443 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:43.941012 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:43.941021 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:43.941401 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 15:42:43.941683 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 15:42:43.942253 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":292,"threads_started":1}
I20260808 15:42:43.945294 31438 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:43.945617 31438 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:43.946453 31438 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:43.948030 30936 ts_manager.cc:194] Registered new tserver with Master: aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:43.948895 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:53611
I20260808 15:42:43.956593 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.130.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.61.keytab.
Entry for principal oryx/127.23.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.61.keytab.
I20260808 15:42:44.016592 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:44743
--webserver_interface=127.23.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.61.keytab
--principal=oryx/127.23.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:46165,127.23.130.61:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:44.361138 31443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:44.361485 31443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:44.361644 31443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:44.368953 31443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:44.369095 31443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:44.369210 31443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:44.369335 31443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:44.369417 31443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:44.379528 31443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:46165,127.23.130.61:44743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.61.keytab
--principal=oryx/127.23.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
W20260808 15:42:44.379913 31443 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.130.62:46165,127.23.130.61:44743'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 15:42:44.380981 31443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:44.382210 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:44.394263 31451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:44.394358 31450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:44.394366 31453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:44.395419 31443 server_base.cc:1034] running on GCE node
I20260808 15:42:44.396157 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:44.397476 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:44.398720 31443 hybrid_clock.cc:648] HybridClock initialized: now 1786203764398628 us; error 54 us; skew 500 ppm
I20260808 15:42:44.399065 31443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203764, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:44.402757 31443 init.cc:377] Logged in from keytab as oryx/127.23.130.61@KRBTEST.COM (short username oryx)
I20260808 15:42:44.405568 31443 webserver.cc:460] Webserver started at http://127.23.130.61:37839/ using document root <none> and password file <none>
I20260808 15:42:44.406558 31443 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:44.406708 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:44.407109 31443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:44.410255 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/instance:
uuid: "e30490cb4fa5493db178256bded15f28"
format_stamp: "Formatted at 2026-08-08 15:42:44 on dist-test-slave-gp6n"
server_key: "5dd761e36c365611752a1b8b25bf0edb"
server_key_iv: "7ef12f0fe65807ae4b926fe965ddf2cf"
server_key_version: "encryptionkey@0"
I20260808 15:42:44.411203 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal/instance:
uuid: "e30490cb4fa5493db178256bded15f28"
format_stamp: "Formatted at 2026-08-08 15:42:44 on dist-test-slave-gp6n"
server_key: "5dd761e36c365611752a1b8b25bf0edb"
server_key_iv: "7ef12f0fe65807ae4b926fe965ddf2cf"
server_key_version: "encryptionkey@0"
I20260808 15:42:44.416684 31443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:42:44.420270 31460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:44.421837 31443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:44.422021 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "e30490cb4fa5493db178256bded15f28"
format_stamp: "Formatted at 2026-08-08 15:42:44 on dist-test-slave-gp6n"
server_key: "5dd761e36c365611752a1b8b25bf0edb"
server_key_iv: "7ef12f0fe65807ae4b926fe965ddf2cf"
server_key_version: "encryptionkey@0"
I20260808 15:42:44.422268 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:44.435027 31292 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
I20260808 15:42:44.443519 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:44.448122 31443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:44.448391 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:44.467947 31443 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.61:44743
I20260808 15:42:44.467960 31518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.61:44743 every 8 connection(s)
I20260808 15:42:44.469369 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
I20260808 15:42:44.472054 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31443
I20260808 15:42:44.473392 31519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:44.483315 31519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203764, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.61@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:44.497995 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.474381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.23.130.61:44743)
0808 15:42:44.475258 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.475275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.475326 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:42:44.477013 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.477040 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.477549 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.477990 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.479498 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.481602 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.483154 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.484219 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.484430 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.489570 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.489577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.489587 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.489621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.493314 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.494341 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.494349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.494358 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.494603 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.495016 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.495021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.495025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.495699 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.495959 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.496549 (+ 590us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.496990 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":345,"threads_started":1}
I20260808 15:42:44.504326 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.486185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.23.130.62:46165)
0808 15:42:44.487637 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.487646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.487671 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:42:44.487742 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.487749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.487869 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.487994 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.489342 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.490110 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.492404 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.492864 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.492987 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.497387 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.497392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.497396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.497430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.501228 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.501987 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.501993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.501998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.502128 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.502627 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.502631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.502634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.502964 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.503147 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.503633 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.503880 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1231,"threads_started":1}
I20260808 15:42:44.504347 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.486144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44810)
0808 15:42:44.487032 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.487088 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.487288 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.488209 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.488214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.488897 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.489154 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.489162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.490254 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.490258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.492183 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.492200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.493066 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.493075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.493281 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.494091 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.494108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.497166 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.501386 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.501399 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.501407 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.501846 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.502251 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.502257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.502262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.502469 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.503107 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.503115 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.503314 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.503514 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.503880 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":364,"threads_started":1}
I20260808 15:42:44.506484 31519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203764, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.61@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
I20260808 15:42:44.519038 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.507312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44248)
0808 15:42:44.507532 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.507553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.507660 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.508024 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.508027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.508357 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.508490 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.508497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.509549 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.509553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.510472 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.510489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.511224 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.511228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.511362 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.511862 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.511877 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.513899 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.516871 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.516875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.516878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.517262 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.517605 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.517609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.517611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.517744 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.518210 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.518213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.518360 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.518557 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.518781 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:42:44.519279 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.507585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.23.130.61:44743)
0808 15:42:44.507737 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.507742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.507765 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:42:44.507816 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.507820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.507869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.507950 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.508641 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.509410 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.510607 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.511090 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.511200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.514060 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.514065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.514068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.514098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.516714 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.517373 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.517376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.517379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.517508 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.517859 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.517862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.517864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.518110 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.518214 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.518665 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.518925 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":58}
I20260808 15:42:44.524207 31519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Bootstrap starting.
I20260808 15:42:44.527417 31519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:44.528430 31519 log.cc:824] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:44.531075 31519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: No bootstrap required, opened a new log
I20260808 15:42:44.537937 31519 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } }
I20260808 15:42:44.538247 31519 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:44.538352 31519 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30490cb4fa5493db178256bded15f28, State: Initialized, Role: FOLLOWER
I20260808 15:42:44.538944 31519 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } }
I20260808 15:42:44.539776 31529 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } } }
I20260808 15:42:44.540079 31529 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:44.540727 31519 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:42:44.550014 31542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:42:44.550192 31542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:42:44.553221 31443 master_runner.cc:428] Detected that this master e30490cb4fa5493db178256bded15f28 is joining an existing cluster
I20260808 15:42:44.553332 31443 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.130.61:44743
I20260808 15:42:44.583165 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.558305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.23.130.62:46165)
0808 15:42:44.558544 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.558553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.567034 (+ 8481us) server_negotiation.cc:407] Connection header received
0808 15:42:44.567365 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.567371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.567505 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.567635 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.569986 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.571015 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.573045 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.573662 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.573781 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.576368 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.576373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.576378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.576417 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.580099 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.580876 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.580883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.580889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.581053 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.581476 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.581483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.581487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.581852 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.582020 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.582529 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.582779 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:42:44.583221 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.557734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.23.130.61:44743)
0808 15:42:44.558015 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.558026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.567469 (+ 9443us) server_negotiation.cc:407] Connection header received
0808 15:42:44.567646 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.567654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.567738 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.568144 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.569549 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.570633 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.572746 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.573317 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.573391 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.576409 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.576413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.576417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.576445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.579614 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.580355 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.580364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.580370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.580503 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.580953 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.580958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.580962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.581351 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.581538 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.582588 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.582802 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":93}
I20260808 15:42:44.583276 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.557830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44820)
0808 15:42:44.566883 (+ 9053us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.566914 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.567184 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.568364 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.568371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.569368 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.569807 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.569822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.571167 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.571173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.572901 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.572913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.573852 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.573860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.574088 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.574700 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.574721 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.576151 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.580234 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.580239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.580244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.580747 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.581189 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.581194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.581197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.581351 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.581976 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.581979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.582138 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.582304 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.582646 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8873,"mutex_wait_us":4,"thread_start_us":8710,"threads_started":1}
I20260808 15:42:44.584182 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.557969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44254)
0808 15:42:44.567372 (+ 9403us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.567387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.567528 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.568167 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.568173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.568914 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.569377 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.569393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.570818 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.570823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.572573 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.572588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.573618 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.573623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.573771 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.574557 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.574598 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.576191 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.579799 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.579805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.579809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.580220 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.580650 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.580658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.580665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.580852 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.582099 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.582104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.582279 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.583430 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.583740 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9255,"mutex_wait_us":57,"spinlock_wait_cycles":26496,"thread_start_us":9032,"threads_started":1}
WARNING: no policy specified for oryx/127.23.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.60@KRBTEST.COM" created.
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203762, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.62@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
I20260808 15:42:44.610514 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.591359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.23.130.61:44743)
0808 15:42:44.591555 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.591563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.592233 (+ 670us) server_negotiation.cc:407] Connection header received
0808 15:42:44.592603 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.592609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.592715 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.592787 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.594521 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.595637 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.597609 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.598395 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.598519 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.602990 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.602996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.603001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.603037 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.606957 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.608026 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.608032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.608036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.608191 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.608690 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.608700 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.608704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.609090 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.609240 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.609862 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.610084 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:42:44.610818 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.591242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44262)
0808 15:42:44.592101 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.592153 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.592381 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.593085 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.593093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.594001 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.594315 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.594329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.595815 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.595821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.597437 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.597452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.598644 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.598653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.598923 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.599695 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.599728 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.602769 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.607165 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.607181 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.607193 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.607892 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.608327 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.608335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.608343 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.608555 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.609248 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.609260 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.609514 (+ 254us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.609755 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.610199 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":440,"threads_started":1}
I20260808 15:42:44.613220 30936 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.130.61:44743: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e30490cb4fa5493db178256bded15f28"
member_type: NON_VOTER
last_known_addr {
host: "127.23.130.61"
port: 44743
}
attrs {
promote: true
}
}
dest_uuid: "c35d5c8533344b349a964b9002cf0401"
cas_config_opid_index: -1
I20260808 15:42:44.614810 30936 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } }
I20260808 15:42:44.620255 31558 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e30490cb4fa5493db178256bded15f28 (127.23.130.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } }
I20260808 15:42:44.621131 31489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:44.625728 31489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Refusing update from remote peer c35d5c8533344b349a964b9002cf0401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.60.keytab.
Entry for principal oryx/127.23.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.60.keytab.
I20260808 15:42:44.626968 31558 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.130.61:44743
I20260808 15:42:44.627874 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:46375
--webserver_interface=127.23.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.60.keytab
--principal=oryx/127.23.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:42:44.628032 31443 master.cc:537] Master@127.23.130.61:44743 shutting down...
I20260808 15:42:44.627810 31557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } }
I20260808 15:42:44.628198 31557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:44.632262 31557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:42:44.635854 30923 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 15:42:44.642637 30923 consensus_peers.cc:487] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 -> Peer e30490cb4fa5493db178256bded15f28 (127.23.130.61:44743): Couldn't send request to peer e30490cb4fa5493db178256bded15f28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:42:44.644793 31443 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:44.645260 31443 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:44.645473 31443 tablet_replica.cc:330] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: stopping tablet replica
I20260808 15:42:44.653080 31443 master.cc:552] Master@127.23.130.61:44743 shutdown complete.
I20260808 15:42:44.653225 31443 master_runner.cc:305] Clearing existing system tablet
I20260808 15:42:44.653709 31443 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:44.653807 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:44.657066 31443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 15:42:44.658895 31565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:44.659267 31443 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:44.659548 31443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:44.659631 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "e30490cb4fa5493db178256bded15f28"
format_stamp: "Formatted at 2026-08-08 15:42:44 on dist-test-slave-gp6n"
server_key: "5dd761e36c365611752a1b8b25bf0edb"
server_key_iv: "7ef12f0fe65807ae4b926fe965ddf2cf"
server_key_version: "encryptionkey@0"
I20260808 15:42:44.664518 31443 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:44.671638 31443 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:42:44.671834 31443 log.cc:1196] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 15:42:44.672274 31443 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Deleting consensus metadata
I20260808 15:42:44.672580 31443 master_runner.cc:315] Copying system tablet from 127.23.130.62:46165
I20260808 15:42:44.682749 31443 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: tablet copy: Beginning tablet copy session from remote peer at address 127.23.130.62:46165
I20260808 15:42:44.699702 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.683602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.23.130.62:46165)
0808 15:42:44.683808 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:44.683815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:44.684437 (+ 622us) server_negotiation.cc:407] Connection header received
0808 15:42:44.684679 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:44.684686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:44.684772 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:44.684879 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:44.686161 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.687287 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.689241 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.689935 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.690034 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.692647 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:44.692654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:44.692659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:44.692701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:44.696319 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.697194 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.697201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.697205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.697384 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:44.697845 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:44.697851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:44.697855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:44.698203 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:44.698342 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:44.698934 (+ 592us) server_negotiation.cc:299] Negotiation successful
0808 15:42:44.699257 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:42:44.700835 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:44.683530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44824)
0808 15:42:44.684232 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:44.684255 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:44.684501 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:44.684943 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:44.684948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:44.685600 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:44.685900 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.685914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.687478 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.687484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.689068 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:44.689084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:44.690322 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:44.690328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:44.690568 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:44.691317 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:44.691342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:44.692419 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:44.696554 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.696559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.696564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.697048 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.697536 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:44.697542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:44.697546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:44.697714 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:44.698470 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:44.698475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:44.698647 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 15:42:44.699892 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0808 15:42:44.700312 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":446,"threads_started":1}
I20260808 15:42:44.701779 30957 tablet_copy_service.cc:140] P c35d5c8533344b349a964b9002cf0401: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e30490cb4fa5493db178256bded15f28 ({username='oryx', principal='oryx/127.23.130.61@KRBTEST.COM'} at 127.0.0.1:44824)
I20260808 15:42:44.702092 30957 tablet_copy_service.cc:161] P c35d5c8533344b349a964b9002cf0401: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e30490cb4fa5493db178256bded15f28 at {username='oryx', principal='oryx/127.23.130.61@KRBTEST.COM'} at 127.0.0.1:44824: session id = e30490cb4fa5493db178256bded15f28-00000000000000000000000000000000
I20260808 15:42:44.707370 30957 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:42:44.709672 31443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:42:44.716495 31443 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: tablet copy: Starting download of 0 data blocks...
I20260808 15:42:44.716912 31443 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: tablet copy: Starting download of 1 WAL segments...
I20260808 15:42:44.719623 31443 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:42:44.724162 30957 tablet_copy_service.cc:342] P c35d5c8533344b349a964b9002cf0401: Request end of tablet copy session e30490cb4fa5493db178256bded15f28-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.130.61@KRBTEST.COM'} at 127.0.0.1:44824
I20260808 15:42:44.724380 30957 tablet_copy_service.cc:433] P c35d5c8533344b349a964b9002cf0401: ending tablet copy session e30490cb4fa5493db178256bded15f28-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e30490cb4fa5493db178256bded15f28
I20260808 15:42:44.736709 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:44.739691 31572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:44.740296 31573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:44.748479 31443 server_base.cc:1034] running on GCE node
W20260808 15:42:44.749200 31575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:44.749687 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:44.750443 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:44.751706 31443 hybrid_clock.cc:648] HybridClock initialized: now 1786203764751637 us; error 42 us; skew 500 ppm
I20260808 15:42:44.751957 31443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:44 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203764, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:44.754143 31443 init.cc:377] Logged in from keytab as oryx/127.23.130.61@KRBTEST.COM (short username oryx)
I20260808 15:42:44.759424 31443 webserver.cc:460] Webserver started at http://127.23.130.61:41767/ using document root <none> and password file <none>
I20260808 15:42:44.759800 31443 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:44.759884 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:44.763163 31443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 15:42:44.765194 31583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:44.765646 31443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260808 15:42:44.765830 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "e30490cb4fa5493db178256bded15f28"
format_stamp: "Formatted at 2026-08-08 15:42:44 on dist-test-slave-gp6n"
server_key: "5dd761e36c365611752a1b8b25bf0edb"
server_key_iv: "7ef12f0fe65807ae4b926fe965ddf2cf"
server_key_version: "encryptionkey@0"
I20260808 15:42:44.766062 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:44.782733 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:44.786799 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:44.817580 31443 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.61:44743
I20260808 15:42:44.817652 31638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.61:44743 every 8 connection(s)
I20260808 15:42:44.818846 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
I20260808 15:42:44.824913 31641 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:42:44.829739 31641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Bootstrap starting.
I20260808 15:42:44.848824 31641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:42:44.849429 31641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: Bootstrap complete.
I20260808 15:42:44.850590 31641 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } }
I20260808 15:42:44.850862 31641 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e30490cb4fa5493db178256bded15f28, State: Initialized, Role: LEARNER
I20260808 15:42:44.851186 31641 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } }
I20260808 15:42:44.852041 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } }
I20260808 15:42:44.852325 31641 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:44.852418 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:42:44.866882 31658 catalog_manager.cc:1259] Loaded cluster ID: 68cf3bdc3afd467b817b4137a427bcf7
I20260808 15:42:44.867012 31658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: loading cluster ID for follower catalog manager: success
I20260808 15:42:44.870522 31658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: acquiring CA information for follower catalog manager: success
I20260808 15:42:44.873039 31658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:42:44.951625 31438 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
W20260808 15:42:45.082168 31561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:45.082625 31561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:45.082803 31561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:45.090376 31561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:45.090551 31561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:45.090641 31561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:45.090754 31561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:45.090831 31561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:45.104465 31561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.60.keytab
--principal=oryx/127.23.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:46375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:45.105865 31561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:45.107321 31561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:45.119417 31664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.119477 31663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.119441 31667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:45.120103 31561 server_base.cc:1034] running on GCE node
I20260808 15:42:45.120936 31561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:45.122538 31561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:45.123760 31561 hybrid_clock.cc:648] HybridClock initialized: now 1786203765123725 us; error 43 us; skew 500 ppm
I20260808 15:42:45.124112 31561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:45.127476 31561 init.cc:377] Logged in from keytab as oryx/127.23.130.60@KRBTEST.COM (short username oryx)
I20260808 15:42:45.128881 31561 webserver.cc:460] Webserver started at http://127.23.130.60:45833/ using document root <none> and password file <none>
I20260808 15:42:45.129530 31561 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:45.129633 31561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:45.129891 31561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:45.133206 31561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/instance:
uuid: "6740ffbd9db54408a62339daaa5773de"
format_stamp: "Formatted at 2026-08-08 15:42:45 on dist-test-slave-gp6n"
server_key: "f2dcb7c64649085ffd721e50125aea30"
server_key_iv: "dff5dd7cf80ee9bedcd4cc3de60d3325"
server_key_version: "encryptionkey@0"
I20260808 15:42:45.133986 31561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal/instance:
uuid: "6740ffbd9db54408a62339daaa5773de"
format_stamp: "Formatted at 2026-08-08 15:42:45 on dist-test-slave-gp6n"
server_key: "f2dcb7c64649085ffd721e50125aea30"
server_key_iv: "dff5dd7cf80ee9bedcd4cc3de60d3325"
server_key_version: "encryptionkey@0"
I20260808 15:42:45.139197 31561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:42:45.142613 31674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:45.144258 31561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:42:45.144444 31561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "6740ffbd9db54408a62339daaa5773de"
format_stamp: "Formatted at 2026-08-08 15:42:45 on dist-test-slave-gp6n"
server_key: "f2dcb7c64649085ffd721e50125aea30"
server_key_iv: "dff5dd7cf80ee9bedcd4cc3de60d3325"
server_key_version: "encryptionkey@0"
I20260808 15:42:45.144673 31561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:45.169991 31561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:45.174259 31561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:45.174598 31561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:45.195405 31561 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.60:46375
I20260808 15:42:45.195499 31729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.60:46375 every 8 connection(s)
I20260808 15:42:45.196802 31561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
I20260808 15:42:45.200765 31730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:45.205864 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31561
I20260808 15:42:45.208288 31730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.60@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
I20260808 15:42:45.228370 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.211160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.23.130.62:46165)
0808 15:42:45.212436 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.212443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.212463 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:42:45.212531 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.212537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.212616 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.212740 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.213977 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.214740 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.216464 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.216981 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.217076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.220874 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.220883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.220892 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.220945 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.224272 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.225275 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.225279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.225282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.225409 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.225998 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.226004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.226007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.226507 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.226647 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.227713 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.227946 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":1045,"threads_started":1}
I20260808 15:42:45.230887 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.211050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44838)
0808 15:42:45.211845 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.211899 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.212071 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.212795 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.212800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.213387 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.213787 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.213816 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.214877 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.214882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.216329 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.216341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.217232 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.217240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.217494 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.218365 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.218392 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.220643 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.224421 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.224433 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.224445 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.225138 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.225533 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.225538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.225542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.225842 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.226763 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.226771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.227239 (+ 468us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.228903 (+ 1664us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.229434 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":411,"threads_started":1}
I20260808 15:42:45.231848 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.208132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.23.130.60:46375)
0808 15:42:45.208867 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.208878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.208927 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:42:45.210760 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.210781 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.211267 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.211795 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.213820 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.215257 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.216770 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.217598 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.217758 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.223084 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.223099 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.223117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.223167 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.227367 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.228626 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.228637 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.228641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.228786 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.229241 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.229246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.229249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.229671 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.229955 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.231172 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.231411 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260808 15:42:45.232962 31730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:42:45.239946 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.227417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44278)
0808 15:42:45.231869 (+ 4452us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.231898 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.232053 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.233005 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.233010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.233037 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:45.233282 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.233293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.235462 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.235490 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.237535 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.237547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.239082 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.239088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.239291 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.239406 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.239498 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.239666 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4298,"thread_start_us":4160,"threads_started":1}
I20260808 15:42:45.240098 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.227585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.23.130.61:44743)
0808 15:42:45.232642 (+ 5057us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.232649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.232667 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:42:45.232735 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.232739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.232834 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.232963 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:45.233473 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.235281 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.237694 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.238886 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.239029 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.239174 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.239539 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.239684 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4909,"thread_start_us":4772,"threads_started":1}
I20260808 15:42:45.241037 31611 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.60@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
I20260808 15:42:45.241658 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35d5c8533344b349a964b9002cf0401. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } }
I20260808 15:42:45.241967 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:42:45.247617 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.233854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44288)
0808 15:42:45.234129 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.234147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.234235 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.235169 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.235174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.235610 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.235739 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.235747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.237209 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.237212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.238525 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.238540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.239488 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.239494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.239670 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.240307 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.240325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.242471 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.245409 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.245413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.245417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.245786 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.246188 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.246193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.246197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.246347 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.246942 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.246946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.247069 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.247178 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.247381 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260808 15:42:45.247695 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.233915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.23.130.61:44743)
0808 15:42:45.234842 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.234847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.234864 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:45.234909 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.234913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.234974 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.235117 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.235898 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.237086 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.238663 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.239296 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.239420 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.242670 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.242675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.242678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.242708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.245270 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.245911 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.245916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.245919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.246073 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.246480 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.246487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.246490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.246813 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.246986 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.247278 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.247442 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":642,"threads_started":1}
I20260808 15:42:45.248770 31730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.60@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
I20260808 15:42:45.262885 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.249518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37498)
0808 15:42:45.249686 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.249703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.249802 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.250291 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.250295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.250826 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.251012 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.251020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.252177 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.252184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.253082 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.253093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.253769 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.253774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.254045 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.254539 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.254562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.256660 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.260657 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.260661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.260663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.261000 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.261391 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.261394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.261396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.261551 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.262136 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.262139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.262263 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.262367 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.262606 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":7}
I20260808 15:42:45.263171 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.249792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.23.130.60:46375)
0808 15:42:45.249966 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.249971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.249989 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:42:45.250103 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.250108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.250173 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.250276 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.251138 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.252057 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.253227 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.253629 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.253762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.256901 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.256906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.256910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.256969 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.260529 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.261109 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.261115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.261119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.261269 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.261654 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.261659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.261663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.262029 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.262183 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.262516 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.262796 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":50}
I20260808 15:42:45.268049 31730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Bootstrap starting.
I20260808 15:42:45.271552 31730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:45.272576 31730 log.cc:824] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:45.275262 31730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: No bootstrap required, opened a new log
I20260808 15:42:45.282184 31730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } }
I20260808 15:42:45.282557 31730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:45.282681 31730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6740ffbd9db54408a62339daaa5773de, State: Initialized, Role: FOLLOWER
I20260808 15:42:45.283269 31730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } }
I20260808 15:42:45.284116 31745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } } }
I20260808 15:42:45.284406 31745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:45.285100 31730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:42:45.295660 31756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:42:45.295822 31756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:42:45.299962 31561 master_runner.cc:428] Detected that this master 6740ffbd9db54408a62339daaa5773de is joining an existing cluster
I20260808 15:42:45.300129 31561 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.130.60:46375
I20260808 15:42:45.326445 31745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:45.326828 31745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } }
I20260808 15:42:45.328681 31745 leader_election.cc:288] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165), e30490cb4fa5493db178256bded15f28 (127.23.130.61:44743)
I20260808 15:42:45.329651 30947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6740ffbd9db54408a62339daaa5773de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d5c8533344b349a964b9002cf0401" is_pre_election: true
I20260808 15:42:45.330595 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.305103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.23.130.62:46165)
0808 15:42:45.305361 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.305368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.311733 (+ 6365us) server_negotiation.cc:407] Connection header received
0808 15:42:45.312294 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.312301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.312390 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.312467 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.314428 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.315946 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.318345 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.319035 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.319129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.322076 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.322083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.322088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.322131 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.325984 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.327995 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.328001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.328004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.328178 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.328703 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.328712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.328718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.329151 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.329309 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.329970 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.330217 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:42:45.330866 31611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6740ffbd9db54408a62339daaa5773de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30490cb4fa5493db178256bded15f28" is_pre_election: true
I20260808 15:42:45.332254 31679 leader_election.cc:302] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6740ffbd9db54408a62339daaa5773de; no voters: c35d5c8533344b349a964b9002cf0401, e30490cb4fa5493db178256bded15f28
I20260808 15:42:45.333108 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.304288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44850)
0808 15:42:45.311623 (+ 7335us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.311644 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.312122 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.312706 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.312713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.313533 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.314178 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.314199 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.316192 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.316196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.318179 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.318196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.319465 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.319477 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.319753 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.320565 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.320589 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.321855 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.326517 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.326520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.326523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.327852 (+ 1329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.328336 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.328340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.328343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.328545 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.329416 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.329420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.329651 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.330838 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.332588 (+ 1750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7188,"mutex_wait_us":22,"thread_start_us":11861,"threads_started":1}
I20260808 15:42:45.333114 31745 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:42:45.336241 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.304247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.23.130.61:44743)
0808 15:42:45.304835 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.304845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.316337 (+ 11492us) server_negotiation.cc:407] Connection header received
0808 15:42:45.316673 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.316679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.316766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.316839 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.318575 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.319762 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.322011 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.322704 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.322805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.325554 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.325561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.325566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.325609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.330699 (+ 5090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.333777 (+ 3078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.333782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.333786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.333943 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.334445 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.334462 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.334466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.334867 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.335000 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.335659 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.335902 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260808 15:42:45.336416 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.304802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.23.130.60:46375)
0808 15:42:45.304999 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.305005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.315760 (+ 10755us) server_negotiation.cc:407] Connection header received
0808 15:42:45.316609 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.316615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.316694 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.316754 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.318312 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.319436 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.321633 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.322441 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.322542 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.325601 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.325608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.325613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.325655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.332850 (+ 7195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.334176 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.334182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.334187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.334327 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.334725 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.334730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.334733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.335127 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.335275 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.335762 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.335990 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":53}
I20260808 15:42:45.337545 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.304422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37510)
0808 15:42:45.315649 (+ 11227us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.315669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.316440 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.316944 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.316948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.317777 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.318137 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.318150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.319663 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.319668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.321481 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.321496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.322897 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.322906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.323125 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.323985 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.324017 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.325388 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.333069 (+ 7681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.333073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.333078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.334069 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.334481 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.334485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.334488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.334616 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.335352 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.335356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.335522 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.336720 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.337048 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11078,"mutex_wait_us":24,"thread_start_us":11484,"threads_started":1}
I20260808 15:42:45.337716 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.304655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44290)
0808 15:42:45.316198 (+ 11543us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.316230 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.316495 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.317054 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.317059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.317930 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.318364 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.318376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.320005 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.320011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.321832 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.321847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.323171 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.323178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.323363 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.323958 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.323980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.325320 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.330857 (+ 5537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.330863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.330866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.333507 (+ 2641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.334082 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.334088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.334094 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.334279 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.335103 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.335109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.335333 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.336768 (+ 1435us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.337209 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11413,"mutex_wait_us":160,"spinlock_wait_cycles":21888,"thread_start_us":7097,"threads_started":1}
I20260808 15:42:45.339344 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31001
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203762, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.62@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.1@KRBTEST.COM
I20260808 15:42:45.357547 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.340137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37512)
0808 15:42:45.340366 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.340387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.340501 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.341185 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.341192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.341570 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.341704 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.341712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.343267 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.343274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.344718 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.344736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.345897 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.345904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.346112 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.346849 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.346873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.349761 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.354286 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.354294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.354298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.354930 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.355365 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.355371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.355374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.355537 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.356332 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.356337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.356528 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.356698 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.357066 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:42:45.357570 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.340515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.23.130.60:46375)
0808 15:42:45.340881 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.340885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.340896 (+ 11us) server_negotiation.cc:407] Connection header received
0808 15:42:45.340970 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.340973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.341043 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.341151 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.341863 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.343107 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.344923 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.345705 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.346492 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.350009 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.350017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.350023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.350072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.354127 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.355064 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.355069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.355073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.355223 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.355756 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.355761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.355765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.356164 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.356314 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.356890 (+ 576us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.357132 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260808 15:42:45.359771 30936 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.130.60:46375: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6740ffbd9db54408a62339daaa5773de"
member_type: NON_VOTER
last_known_addr {
host: "127.23.130.60"
port: 46375
}
attrs {
promote: true
}
}
dest_uuid: "c35d5c8533344b349a964b9002cf0401"
cas_config_opid_index: 5
I20260808 15:42:45.360929 30936 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } }
I20260808 15:42:45.361204 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.340722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.23.130.1:40311)
0808 15:42:45.341371 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.341381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.341410 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:42:45.341477 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.341485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.341772 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.341951 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.343591 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.344740 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.346215 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.347029 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.347179 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.353330 (+ 6151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.353347 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.353370 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.353416 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.357649 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.358667 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.358676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.358679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.358855 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.359316 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.359321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.359324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.359775 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.360110 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.360501 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.360781 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":317,"threads_started":1}
I20260808 15:42:45.362852 31139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:45.362900 31739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6740ffbd9db54408a62339daaa5773de (127.23.130.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.366134 31739 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.130.60:46375
I20260808 15:42:45.366750 31561 master.cc:537] Master@127.23.130.60:46375 shutting down...
I20260808 15:42:45.367266 31611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Refusing update from remote peer c35d5c8533344b349a964b9002cf0401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 15:42:45.370851 31768 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.371603 31768 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:45.371683 31739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 15:42:45.396899 31611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6740ffbd9db54408a62339daaa5773de (127.23.130.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.404260 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: NON_VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: true } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.404588 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:42:45.404834 31561 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:45.405267 31561 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 0 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:45.405503 31561 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: stopping tablet replica
I20260808 15:42:45.405771 31768 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: attempting to promote NON_VOTER e30490cb4fa5493db178256bded15f28 to VOTER
W20260808 15:42:45.406562 30920 consensus_peers.cc:487] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 -> Peer 6740ffbd9db54408a62339daaa5773de (127.23.130.60:46375): Couldn't send request to peer 6740ffbd9db54408a62339daaa5773de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:42:45.407205 31768 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } }
W20260808 15:42:45.409017 31675 connection.cc:441] server connection from 127.0.0.1:37512 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37512 (request call id 3) could send its response
I20260808 15:42:45.409602 31611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 LEARNER]: Refusing update from remote peer c35d5c8533344b349a964b9002cf0401: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 15:42:45.410079 31739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 15:42:45.411929 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.411379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37516)
0808 15:42:45.411608 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.411731 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260808 15:42:45.412171 30920 consensus_peers.cc:487] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 -> Peer 6740ffbd9db54408a62339daaa5773de (127.23.130.60:46375): Couldn't send request to peer 6740ffbd9db54408a62339daaa5773de. Status: Network error: Client connection negotiation failed: client connection to 127.23.130.60:46375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 15:42:45.412407 31739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e30490cb4fa5493db178256bded15f28 (127.23.130.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.415349 31561 master.cc:552] Master@127.23.130.60:46375 shutdown complete.
I20260808 15:42:45.415483 31561 master_runner.cc:305] Clearing existing system tablet
I20260808 15:42:45.415915 31561 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:45.416095 31561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:45.416359 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.416322 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.416778 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:45.416676 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:45.412989 31611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e30490cb4fa5493db178256bded15f28 (127.23.130.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.421478 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.421717 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:45.422212 31561 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 15:42:45.424098 31776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:45.424430 31561 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:45.424746 31561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:45.424875 31561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "6740ffbd9db54408a62339daaa5773de"
format_stamp: "Formatted at 2026-08-08 15:42:45 on dist-test-slave-gp6n"
server_key: "f2dcb7c64649085ffd721e50125aea30"
server_key_iv: "dff5dd7cf80ee9bedcd4cc3de60d3325"
server_key_version: "encryptionkey@0"
I20260808 15:42:45.428594 31561 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:45.433372 31561 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:42:45.433560 31561 log.cc:1196] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 15:42:45.434010 31561 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Deleting consensus metadata
I20260808 15:42:45.434262 31561 master_runner.cc:315] Copying system tablet from 127.23.130.62:46165
I20260808 15:42:45.436905 31561 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: tablet copy: Beginning tablet copy session from remote peer at address 127.23.130.62:46165
I20260808 15:42:45.451903 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.437642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:44858)
0808 15:42:45.439903 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.439924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.440161 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.440530 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.440534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.441023 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:45.441226 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.441235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.442274 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.442279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.443326 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.443338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.444092 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.444099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.444315 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.445118 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:45.445135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:45.446098 (+ 963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:45.449226 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.449231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.449235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.449598 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.449994 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:45.449999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:45.450003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:45.450169 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:45.450796 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:45.450800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:45.450962 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.451134 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.451460 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"thread_start_us":2019,"threads_started":1}
I20260808 15:42:45.451854 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.437758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.23.130.62:46165)
0808 15:42:45.438030 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.438037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.440026 (+ 1989us) server_negotiation.cc:407] Connection header received
0808 15:42:45.440320 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.440330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.440393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.440505 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:45.441376 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.442122 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.443475 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.443911 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.444050 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.446275 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:45.446281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:45.446288 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:45.446324 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:45.449082 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.449723 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.449731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.449735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.449874 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:45.450279 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:45.450285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:45.450289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:45.450667 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:45.450833 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.451230 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.451460 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260808 15:42:45.453011 30957 tablet_copy_service.cc:140] P c35d5c8533344b349a964b9002cf0401: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6740ffbd9db54408a62339daaa5773de ({username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:44858)
I20260808 15:42:45.453202 30957 tablet_copy_service.cc:161] P c35d5c8533344b349a964b9002cf0401: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6740ffbd9db54408a62339daaa5773de at {username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:44858: session id = 6740ffbd9db54408a62339daaa5773de-00000000000000000000000000000000
I20260808 15:42:45.456029 30957 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:42:45.457831 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:42:45.462818 31561 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: tablet copy: Starting download of 0 data blocks...
I20260808 15:42:45.463120 31561 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: tablet copy: Starting download of 1 WAL segments...
I20260808 15:42:45.465433 31561 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:42:45.468879 30957 tablet_copy_service.cc:342] P c35d5c8533344b349a964b9002cf0401: Request end of tablet copy session 6740ffbd9db54408a62339daaa5773de-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:44858
I20260808 15:42:45.469029 30957 tablet_copy_service.cc:433] P c35d5c8533344b349a964b9002cf0401: ending tablet copy session 6740ffbd9db54408a62339daaa5773de-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6740ffbd9db54408a62339daaa5773de
I20260808 15:42:45.479682 31561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:45.482303 31783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.491082 31784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.499996 31786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:45.500243 31561 server_base.cc:1034] running on GCE node
I20260808 15:42:45.500898 31561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:45.501670 31561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:45.503034 31561 hybrid_clock.cc:648] HybridClock initialized: now 1786203765502971 us; error 45 us; skew 500 ppm
I20260808 15:42:45.503365 31561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:45.505726 31561 init.cc:377] Logged in from keytab as oryx/127.23.130.60@KRBTEST.COM (short username oryx)
I20260808 15:42:45.511439 31561 webserver.cc:460] Webserver started at http://127.23.130.60:38227/ using document root <none> and password file <none>
I20260808 15:42:45.511840 31561 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:45.512002 31561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:45.515257 31561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 15:42:45.517347 31794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:45.517822 31561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260808 15:42:45.518008 31561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "6740ffbd9db54408a62339daaa5773de"
format_stamp: "Formatted at 2026-08-08 15:42:45 on dist-test-slave-gp6n"
server_key: "f2dcb7c64649085ffd721e50125aea30"
server_key_iv: "dff5dd7cf80ee9bedcd4cc3de60d3325"
server_key_version: "encryptionkey@0"
I20260808 15:42:45.518272 31561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:45.535282 31561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:45.539608 31561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:45.544106 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31001
I20260808 15:42:45.575158 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:40311
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:46165
--webserver_port=33725
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:42:45.584954 31561 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.60:46375
I20260808 15:42:45.585001 31852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.60:46375 every 8 connection(s)
I20260808 15:42:45.586292 31561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
I20260808 15:42:45.592756 31853 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:42:45.597222 31853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Bootstrap starting.
I20260808 15:42:45.618331 31853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:42:45.618865 31853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Bootstrap complete.
I20260808 15:42:45.619915 31853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } }
I20260808 15:42:45.620213 31853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6740ffbd9db54408a62339daaa5773de, State: Initialized, Role: LEARNER
I20260808 15:42:45.620483 31853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } }
I20260808 15:42:45.621258 31856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.621551 31853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:45.621675 31856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: LEARNER
I20260808 15:42:45.633857 31868 catalog_manager.cc:1259] Loaded cluster ID: 68cf3bdc3afd467b817b4137a427bcf7
I20260808 15:42:45.633968 31868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: loading cluster ID for follower catalog manager: success
I20260808 15:42:45.637303 31868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: acquiring CA information for follower catalog manager: success
I20260808 15:42:45.639845 31868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:42:45.942174 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.936301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.23.130.60:46375)
0808 15:42:45.937028 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:45.937032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:45.937049 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:45.937112 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:45.937115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:45.937190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:45.937322 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:45.937769 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.938803 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.940324 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.941142 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.941300 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.941522 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:45.941700 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 15:42:45.941841 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":412,"threads_started":1}
I20260808 15:42:45.942152 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:45.936149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37524)
0808 15:42:45.936682 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:45.936707 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:45.936865 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:45.937357 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:45.937363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:45.937395 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:45.937610 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.937620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.938953 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.938959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.940168 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:45.940181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:45.941338 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:45.941344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:45.941515 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:45.941555 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:42:45.941673 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:42:45.941819 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":310,"threads_started":1}
I20260808 15:42:45.943531 31824 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 15:42:45.944342 31856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35d5c8533344b349a964b9002cf0401. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:45.944698 31856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: LEARNER
W20260808 15:42:45.962293 31851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:45.962579 31851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:45.962774 31851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:45.970177 31851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:45.970288 31851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:45.970506 31851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:45.980276 31851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:40311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=33725
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:45.981598 31851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:45.982947 31851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:45.996357 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.996446 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:45.996412 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:45.997190 31851 server_base.cc:1034] running on GCE node
I20260808 15:42:45.997783 31851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:45.998839 31851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:46.000085 31851 hybrid_clock.cc:648] HybridClock initialized: now 1786203766000029 us; error 46 us; skew 500 ppm
I20260808 15:42:46.000430 31851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:45 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:46.004172 31851 init.cc:377] Logged in from keytab as oryx/127.23.130.1@KRBTEST.COM (short username oryx)
I20260808 15:42:46.005707 31851 webserver.cc:460] Webserver started at http://127.23.130.1:33725/ using document root <none> and password file <none>
I20260808 15:42:46.006598 31851 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:46.006771 31851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:46.012146 31851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:46.014973 31888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.016363 31851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:42:46.016531 31851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "1681ff6962684d8db69f0265bdd39310"
format_stamp: "Formatted at 2026-08-08 15:42:42 on dist-test-slave-gp6n"
server_key: "8e1745f45bef3fd5cdf2c796c479e527"
server_key_iv: "ba68b1be87cc9ea4f9118f9a80a3ac42"
server_key_version: "encryptionkey@0"
I20260808 15:42:46.017524 31851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:46.050974 31851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:46.055516 31851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:46.055869 31851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:46.056903 31851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:46.058324 31851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:46.058434 31851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.058557 31851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:46.058621 31851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.094558 31851 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:40311
I20260808 15:42:46.094650 32011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:40311 every 8 connection(s)
I20260808 15:42:46.096045 31851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:46.098150 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 31851
I20260808 15:42:46.098515 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31150
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203765, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.2@KRBTEST.COM
I20260808 15:42:46.150027 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.099828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:57695 (local address 127.23.130.60:46375)
0808 15:42:46.100071 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.100081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.123804 (+ 23723us) server_negotiation.cc:407] Connection header received
0808 15:42:46.126224 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.126231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.126334 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.126495 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.130861 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.131969 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.134981 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.135661 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.135764 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.142562 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.142568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.142571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.142604 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.146062 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147081 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.147090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.147241 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147810 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.147818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.148181 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.148325 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.149446 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.149673 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:42:46.150159 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.107179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:46253 (local address 127.23.130.61:44743)
0808 15:42:46.123261 (+ 16082us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.123268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.129125 (+ 5857us) server_negotiation.cc:407] Connection header received
0808 15:42:46.129205 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.129210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.129325 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.129435 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.130820 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.131893 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.135016 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.135717 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.135823 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.141824 (+ 6001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.141831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.141837 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.141880 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.146334 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147287 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.147296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.147458 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147991 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.148001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.148404 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.148550 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.149474 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.149719 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15909,"thread_start_us":16639,"threads_started":1}
I20260808 15:42:46.151437 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.100406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:35027)
0808 15:42:46.124248 (+ 23842us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.124282 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.125885 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.127332 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.127337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.128928 (+ 1591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.130593 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.130607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.132149 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.132155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.134800 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.134814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.136110 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.136117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.136588 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.137596 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.137631 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.141418 (+ 3787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.146103 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.146115 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.146137 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.146931 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.147373 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.147381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.147659 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.148408 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.148416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.148962 (+ 546us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.149320 (+ 358us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.149845 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23662,"mutex_wait_us":1108,"spinlock_wait_cycles":24832,"thread_start_us":23991,"threads_started":1}
I20260808 15:42:46.151495 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.106618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.23.130.1:46253)
0808 15:42:46.128852 (+ 22234us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.128872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.129008 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.129628 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.129632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.130256 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.130670 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.130684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.132106 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.132111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.134839 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.134853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.136183 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.136190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.136574 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.137646 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.137676 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.141576 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.146547 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.146554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.146560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.147131 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.147621 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147653 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.147657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.147845 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.148640 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.148644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.148976 (+ 332us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.150497 (+ 1521us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.150782 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22092,"mutex_wait_us":298,"spinlock_wait_cycles":67072,"thread_start_us":16081,"threads_started":1}
I20260808 15:42:46.151520 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.099780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.23.130.1:57695)
0808 15:42:46.123600 (+ 23820us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.123653 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.125885 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.127294 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.127313 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.128925 (+ 1612us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.130105 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.130134 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.132169 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.132173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.134799 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.134814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.136110 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.136117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.136584 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.137506 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.137547 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.142292 (+ 4745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.146330 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.146335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.146339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.146952 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.147379 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.147386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.147681 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.148408 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.148416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.148962 (+ 546us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.150710 (+ 1748us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.151027 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23539,"mutex_wait_us":591,"spinlock_wait_cycles":68352,"thread_start_us":28087,"threads_started":1}
I20260808 15:42:46.152204 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.099468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.23.130.2:34787)
0808 15:42:46.115152 (+ 15684us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.115164 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.127197 (+ 12033us) server_negotiation.cc:407] Connection header received
0808 15:42:46.127283 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.127294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.127586 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.127748 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.130072 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.131516 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.133569 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.134442 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.134569 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.143624 (+ 9055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.143641 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.143663 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.143715 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.147851 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.148839 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.148848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.148850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.149028 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.149513 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.149518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.149521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.149951 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.150270 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.151473 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.151763 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15462,"thread_start_us":15701,"threads_started":1}
I20260808 15:42:46.152666 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.099122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:35027 (local address 127.23.130.62:46165)
0808 15:42:46.114781 (+ 15659us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.114790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.124461 (+ 9671us) server_negotiation.cc:407] Connection header received
0808 15:42:46.126814 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.126820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.126924 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.127150 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.130797 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.131989 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.134981 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.135661 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.135779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.141682 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.141690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.141696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.141744 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.145924 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147080 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.147089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.147238 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.147806 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.147811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.147814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.148181 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.148326 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.151969 (+ 3643us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.152241 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15498,"thread_start_us":15675,"threads_started":1}
I20260808 15:42:46.153735 31287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:46.153479 32016 heartbeater.cc:347] Connected to a master server at 127.23.130.61:44743
I20260808 15:42:46.154136 32016 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.153725 32014 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:46.154520 32014 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.154734 32012 heartbeater.cc:347] Connected to a master server at 127.23.130.60:46375
I20260808 15:42:46.154891 32012 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.155870 32014 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:46.156035 32016 heartbeater.cc:511] Master 127.23.130.61:44743 requested a full tablet report, sending...
I20260808 15:42:46.155915 32012 heartbeater.cc:511] Master 127.23.130.60:46375 requested a full tablet report, sending...
I20260808 15:42:46.157871 30936 ts_manager.cc:194] Re-registered known tserver with Master: 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311)
I20260808 15:42:46.158355 31813 ts_manager.cc:194] Registered new tserver with Master: 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311)
I20260808 15:42:46.158471 31601 ts_manager.cc:194] Registered new tserver with Master: 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311)
I20260808 15:42:46.159133 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:35027
I20260808 15:42:46.252015 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31150
I20260808 15:42:46.272503 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:34787
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:46165
--webserver_port=38191
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:42:46.381783 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: NON_VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: true } } }
I20260808 15:42:46.382354 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:46.383989 32029 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: attempting to promote NON_VOTER 6740ffbd9db54408a62339daaa5773de to VOTER
I20260808 15:42:46.384584 32029 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } }
I20260808 15:42:46.386866 31824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 LEARNER]: Refusing update from remote peer c35d5c8533344b349a964b9002cf0401: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:42:46.386824 31611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Refusing update from remote peer c35d5c8533344b349a964b9002cf0401: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:42:46.387583 32030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:42:46.388060 32029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:42:46.391286 31739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6740ffbd9db54408a62339daaa5773de (127.23.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.391732 31611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6740ffbd9db54408a62339daaa5773de (127.23.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.394295 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.394593 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:46.396667 32035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.396915 32029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.397090 32035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:46.397186 32029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:46.398133 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.398499 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401 [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:46.398794 31824 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6740ffbd9db54408a62339daaa5773de (127.23.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.401957 31856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d5c8533344b349a964b9002cf0401" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:46.402210 31856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: FOLLOWER
W20260808 15:42:46.620968 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:46.621279 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:46.621476 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:46.629182 32028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:46.629300 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:46.629485 32028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:46.639202 32028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=38191
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:46.640486 32028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:46.641842 32028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:46.656520 32041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:46.656543 32042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:46.656991 32044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:46.657436 32028 server_base.cc:1034] running on GCE node
I20260808 15:42:46.658119 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:46.659235 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:46.660496 32028 hybrid_clock.cc:648] HybridClock initialized: now 1786203766660428 us; error 64 us; skew 500 ppm
I20260808 15:42:46.660856 32028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203766, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:46.664250 32028 init.cc:377] Logged in from keytab as oryx/127.23.130.2@KRBTEST.COM (short username oryx)
I20260808 15:42:46.665637 32028 webserver.cc:460] Webserver started at http://127.23.130.2:38191/ using document root <none> and password file <none>
I20260808 15:42:46.666309 32028 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:46.666525 32028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:46.672096 32028 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 15:42:46.675066 32053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.676501 32028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:46.676640 32028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "24c502541071445aa8c9805607dd731f"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "ae51ce0bebfe33d4ab17c8ec01e21b5d"
server_key_iv: "7e209477b6584d133c3c143dbc7cb266"
server_key_version: "encryptionkey@0"
I20260808 15:42:46.677618 32028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:46.698933 32028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:46.703284 32028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:46.703524 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:46.704412 32028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:46.705760 32028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:46.705848 32028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.705920 32028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:46.705974 32028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:46.741047 32028 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:34787
I20260808 15:42:46.741139 32177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:34787 every 8 connection(s)
I20260808 15:42:46.742513 32028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:46.747758 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 32028
I20260808 15:42:46.748104 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31298
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.3@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203766, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203766, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
Aug 08 15:42:46 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203766, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
I20260808 15:42:46.793797 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.745502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:40205 (local address 127.23.130.62:46165)
0808 15:42:46.769076 (+ 23574us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.769082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.769600 (+ 518us) server_negotiation.cc:407] Connection header received
0808 15:42:46.771960 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.771969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.772133 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.772434 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.775408 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.776402 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.778316 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.778821 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.778947 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.783942 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.783953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.783957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.783991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.788959 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.790749 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.790756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.790761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.790931 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791940 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.791945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.791949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.792287 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.792610 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.793141 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.793387 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23437,"thread_start_us":23675,"threads_started":1}
I20260808 15:42:46.794965 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.749396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.23.130.3:34879)
0808 15:42:46.765728 (+ 16332us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.765740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.768945 (+ 3205us) server_negotiation.cc:407] Connection header received
0808 15:42:46.769224 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.769236 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.769557 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.769715 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.771542 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.773267 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.775163 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.775975 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.776112 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.784591 (+ 8479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.784605 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.784623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.784673 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.788625 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.789541 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.789550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.789554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.789738 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.790208 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.790215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.790218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.791116 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.791513 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.794214 (+ 2701us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.794554 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16097,"thread_start_us":16345,"threads_started":1}
I20260808 15:42:46.795086 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.746203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:40205)
0808 15:42:46.769423 (+ 23220us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.769480 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.771580 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.774553 (+ 2973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.774557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.775025 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.775256 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.775267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.776541 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.776546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.778176 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.778187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.779080 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.779090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.779344 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.780067 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.780106 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.783736 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.789785 (+ 6049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.789792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.789797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.790627 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.791656 (+ 1029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.791663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.791825 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.792413 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.792418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.792684 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.793004 (+ 320us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.793545 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22370,"mutex_wait_us":102,"spinlock_wait_cycles":28416,"thread_start_us":22907,"threads_started":1}
I20260808 15:42:46.796207 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.745945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:35483 (local address 127.23.130.60:46375)
0808 15:42:46.768188 (+ 22243us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.768197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.770227 (+ 2030us) server_negotiation.cc:407] Connection header received
0808 15:42:46.773489 (+ 3262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.773498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.773582 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.773694 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.774957 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.777404 (+ 2447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.779490 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.780135 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.780269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.785939 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.785944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.785947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.785980 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.789611 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.790792 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.790799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.790804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.790953 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791624 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.791629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.791632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.792029 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.792163 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.795318 (+ 3155us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.795577 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22050,"thread_start_us":21900,"threads_started":1}
I20260808 15:42:46.796680 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.745600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:44569 (local address 127.23.130.61:44743)
0808 15:42:46.767698 (+ 22098us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:46.767707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:46.770265 (+ 2558us) server_negotiation.cc:407] Connection header received
0808 15:42:46.772115 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:46.772121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:46.772199 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:46.772333 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:46.774930 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.775904 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.780537 (+ 4633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.781305 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.781417 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.787249 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:46.787257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:46.787263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:46.787324 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:46.790912 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791938 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.791943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.791946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.792096 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:46.793241 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:46.793246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:46.793250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:46.793644 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:46.793809 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:46.794733 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 15:42:46.794949 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21963,"thread_start_us":21831,"threads_started":1}
I20260808 15:42:46.796679 32180 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:46.796859 31432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:46.797019 32180 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.797380 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.746465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.23.130.2:35483)
0808 15:42:46.770057 (+ 23592us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.770075 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.771611 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.773737 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.773742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.774346 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.774767 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.774781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.777540 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.777547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.779323 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.779341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.780601 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.780608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.780802 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.781417 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.781448 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.785747 (+ 4299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.789757 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.789767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.789787 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.790644 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.791096 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.791104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.791456 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.792237 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.792245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.792684 (+ 439us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.793003 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.796733 (+ 3730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22630,"mutex_wait_us":3296,"spinlock_wait_cycles":29696,"thread_start_us":20438,"threads_started":1}
I20260808 15:42:46.797981 32180 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:46.798503 32179 heartbeater.cc:347] Connected to a master server at 127.23.130.60:46375
I20260808 15:42:46.798637 32179 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.799224 32179 heartbeater.cc:511] Master 127.23.130.60:46375 requested a full tablet report, sending...
I20260808 15:42:46.800220 30936 ts_manager.cc:194] Re-registered known tserver with Master: 24c502541071445aa8c9805607dd731f (127.23.130.2:34787)
I20260808 15:42:46.800670 31813 ts_manager.cc:194] Registered new tserver with Master: 24c502541071445aa8c9805607dd731f (127.23.130.2:34787)
I20260808 15:42:46.801525 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:40205
I20260808 15:42:46.802381 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:46.746203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.23.130.2:44569)
0808 15:42:46.770149 (+ 23946us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:46.770169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:46.771961 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:46.772803 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:46.772828 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:46.773757 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:46.774633 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.774663 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.776083 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.776089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.780360 (+ 4271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:46.780375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:46.781846 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:46.781855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:46.782126 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:46.782751 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:46.782774 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:46.787029 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:46.791032 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.791037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.791040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.791528 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.792232 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:46.792238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:46.792241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:46.792397 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:46.794130 (+ 1733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:46.794135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:46.794294 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 15:42:46.797184 (+ 2890us) client_negotiation.cc:240] Negotiation successful
0808 15:42:46.797952 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23773,"mutex_wait_us":2474,"thread_start_us":21346,"threads_started":1}
I20260808 15:42:46.804421 32181 heartbeater.cc:347] Connected to a master server at 127.23.130.61:44743
I20260808 15:42:46.804564 32181 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:46.804919 32181 heartbeater.cc:511] Master 127.23.130.61:44743 requested a full tablet report, sending...
I20260808 15:42:46.805927 31601 ts_manager.cc:194] Registered new tserver with Master: 24c502541071445aa8c9805607dd731f (127.23.130.2:34787)
I20260808 15:42:46.884269 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31298
I20260808 15:42:46.904863 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:34879
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:46165
--webserver_port=42435
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:42:47.161413 32014 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
W20260808 15:42:47.250011 32194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:47.250324 32194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:47.250536 32194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:47.260285 32194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:47.260443 32194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:47.260653 32194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:47.271989 32194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=42435
--tserver_master_addrs=127.23.130.62:46165,127.23.130.61:44743,127.23.130.60:46375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:47.273412 32194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:47.274979 32194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:47.290316 32200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:47.290459 32199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:47.290277 32202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:47.291280 32194 server_base.cc:1034] running on GCE node
I20260808 15:42:47.292009 32194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:47.293190 32194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:47.294497 32194 hybrid_clock.cc:648] HybridClock initialized: now 1786203767294452 us; error 44 us; skew 500 ppm
I20260808 15:42:47.294888 32194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:47 dist-test-slave-gp6n krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203767, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:47.298628 32194 init.cc:377] Logged in from keytab as oryx/127.23.130.3@KRBTEST.COM (short username oryx)
I20260808 15:42:47.300659 32194 webserver.cc:460] Webserver started at http://127.23.130.3:42435/ using document root <none> and password file <none>
I20260808 15:42:47.301470 32194 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:47.301633 32194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:47.307862 32194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:42:47.310897 32212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:47.312350 32194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:42:47.312546 32194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "aa0c4cb87e7547c5b08393456f710052"
format_stamp: "Formatted at 2026-08-08 15:42:43 on dist-test-slave-gp6n"
server_key: "c1a0c57f4fa473e8f47f2ec4392dbbd0"
server_key_iv: "742d62c9116fc51c6fe8871777c4c34f"
server_key_version: "encryptionkey@0"
I20260808 15:42:47.313614 32194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:47.330160 32194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:47.334203 32194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:47.334574 32194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:47.335556 32194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:42:47.336968 32194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:47.337095 32194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:47.337234 32194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:47.337329 32194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:47.372548 32194 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:34879
I20260808 15:42:47.372639 32334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:34879 every 8 connection(s)
I20260808 15:42:47.374048 32194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:47.380699 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 32194
Aug 08 15:42:47 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203767, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.61@KRBTEST.COM
Aug 08 15:42:47 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203767, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.60@KRBTEST.COM
Aug 08 15:42:47 dist-test-slave-gp6n krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203767, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:47.409178 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.378916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.23.130.3:57601)
0808 15:42:47.382895 (+ 3979us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.382920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.385407 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.386377 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.386393 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.387358 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:47.388454 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.388486 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.390099 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.390105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.392540 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.392554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.393837 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.393844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.394131 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.395167 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:47.395204 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:47.398581 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:47.403209 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.403222 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.403246 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.404048 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.404578 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.404585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.404590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.404908 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.405827 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:47.405839 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:47.406426 (+ 587us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.406793 (+ 367us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.407424 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3805,"mutex_wait_us":184,"spinlock_wait_cycles":15616,"thread_start_us":4578,"threads_started":1}
I20260808 15:42:47.410461 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.378469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:57601 (local address 127.23.130.61:44743)
0808 15:42:47.385324 (+ 6855us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.385330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.385354 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:42:47.385818 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.385827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.385939 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.386819 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.388670 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.389886 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.392728 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.393411 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.393544 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.398845 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.398853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.398858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.398907 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.402994 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.404228 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.404236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.404243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.404436 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.405089 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.405097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.405103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.405551 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.405731 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.409720 (+ 3989us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.410012 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6734,"thread_start_us":6626,"threads_started":1}
I20260808 15:42:47.412987 32337 heartbeater.cc:347] Connected to a master server at 127.23.130.61:44743
I20260808 15:42:47.413307 32337 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:47.413982 32337 heartbeater.cc:511] Master 127.23.130.61:44743 requested a full tablet report, sending...
I20260808 15:42:47.416196 31601 ts_manager.cc:194] Registered new tserver with Master: aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:47.418594 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.377293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35051 (local address 127.23.130.60:46375)
0808 15:42:47.386139 (+ 8846us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.386148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.386167 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:47.386225 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.386230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.386308 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.388614 (+ 2306us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.388807 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.389949 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.393179 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.394081 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.394210 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.399425 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.399431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.399435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.399477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.404161 (+ 4684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.405139 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.405145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.405148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.405307 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.414909 (+ 9602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.414919 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.414925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.415447 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.415641 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.417484 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.417978 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8703,"thread_start_us":8573,"threads_started":1}
I20260808 15:42:47.420611 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.378079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.23.130.3:35051)
0808 15:42:47.379295 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.379363 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.385407 (+ 6044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.386531 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.386536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.387347 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:47.388459 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.388486 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.390113 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.390117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.392945 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.392972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.394567 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.394573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.394822 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.395577 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:47.395610 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:47.399195 (+ 3585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:47.404381 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.404386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.404389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.404987 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.405590 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.405598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.405602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.405780 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.416536 (+ 10756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:47.416544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:47.416970 (+ 426us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.419227 (+ 2257us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.419790 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":436,"thread_start_us":4332,"threads_started":1}
I20260808 15:42:47.422643 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.378080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:40617)
0808 15:42:47.386667 (+ 8587us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.386691 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.386872 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.391974 (+ 5102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.391979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.392667 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:47.393427 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.393441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.395279 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.395283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.396990 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.397011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.398111 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.398122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.398304 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.399426 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:47.399461 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:47.409527 (+ 10066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:47.419558 (+ 10031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.419565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.419570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.420188 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.420597 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:47.420604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:47.420607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:47.420789 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:47.421604 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:47.421611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:47.421771 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.421935 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.422229 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8390,"mutex_wait_us":5637,"thread_start_us":2551,"threads_started":1}
I20260808 15:42:47.422711 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.377237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:40617 (local address 127.23.130.62:46165)
0808 15:42:47.391142 (+ 13905us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.391149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.391169 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:42:47.391231 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.391235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.391319 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.391443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.393615 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.395075 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.397164 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.397725 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.397823 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.409952 (+ 12129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.409959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.409964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.410004 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.419426 (+ 9422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.420291 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.420297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.420300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.420468 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.421038 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.421045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.421050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.421474 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.421651 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.422042 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.422255 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13780,"thread_start_us":13679,"threads_started":1}
I20260808 15:42:47.422940 32335 heartbeater.cc:347] Connected to a master server at 127.23.130.60:46375
I20260808 15:42:47.423101 32335 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:47.423599 32335 heartbeater.cc:511] Master 127.23.130.60:46375 requested a full tablet report, sending...
I20260808 15:42:47.423974 32336 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46165
I20260808 15:42:47.424108 32336 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:47.424600 32336 heartbeater.cc:511] Master 127.23.130.62:46165 requested a full tablet report, sending...
I20260808 15:42:47.425431 31813 ts_manager.cc:194] Registered new tserver with Master: aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:47.426035 30936 ts_manager.cc:194] Re-registered known tserver with Master: aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:47.427208 30936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:40617
I20260808 15:42:47.440272 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.409606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.23.130.60:46375)
0808 15:42:47.412406 (+ 2800us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.412412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.418542 (+ 6130us) server_negotiation.cc:407] Connection header received
0808 15:42:47.419018 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.419023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.419107 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.419191 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.424340 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.425498 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.428497 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.429148 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.429241 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.433066 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.433075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.433081 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.433154 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.437049 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.437944 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.437951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.437957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.438132 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.438644 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.438650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.438656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.438993 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.439256 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.439607 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.439844 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2540,"threads_started":1}
I20260808 15:42:47.440608 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.406273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.23.130.61:44743)
0808 15:42:47.411052 (+ 4779us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.411059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.420092 (+ 9033us) server_negotiation.cc:407] Connection header received
0808 15:42:47.420727 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.420732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.420816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.420888 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.426272 (+ 5384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.427394 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.429594 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.430231 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.430334 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.434064 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.434072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.434077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.434124 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.437502 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.438219 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.438224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.438228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.438368 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.438921 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.438928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.438935 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.439331 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.439577 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.439969 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.440219 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4637,"thread_start_us":10878,"threads_started":1}
I20260808 15:42:47.442782 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.400593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.23.130.62:46165)
0808 15:42:47.423345 (+ 22752us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.423352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.423373 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:42:47.423429 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.423433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.423732 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.428120 (+ 4388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.428527 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.429561 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.431779 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.432551 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.432694 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.435873 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.435880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.435885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.435922 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.439532 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.440451 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.440456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.440460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.440720 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.441208 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.441215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.441218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.441562 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.441826 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.442185 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.442445 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22037,"thread_start_us":21891,"threads_started":1}
I20260808 15:42:47.446924 30936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:47.454105 30936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:47.485662 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.474288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40311 (local address 127.0.0.1:47434)
0808 15:42:47.475079 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.475098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.475173 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.476495 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.476505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.476541 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:47.477032 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.477048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.480147 (+ 3099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.480153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.482687 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.482704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.484602 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.484608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.484811 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.484895 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.484993 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.485141 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":69,"thread_start_us":994,"threads_started":1}
I20260808 15:42:47.486104 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.475112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.23.130.1:40311)
0808 15:42:47.475696 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.475707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.475737 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:42:47.475856 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.475865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.476252 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.476471 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:47.478037 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.479987 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.482911 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.484357 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.484512 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.484731 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.485605 (+ 874us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.485801 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"spinlock_wait_cycles":1792,"thread_start_us":314,"threads_started":1}
I20260808 15:42:47.486958 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.473874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34787 (local address 127.0.0.1:36170)
0808 15:42:47.474637 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.474654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.474881 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.479308 (+ 4427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.479314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.479338 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:47.479703 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.479712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.481521 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.481527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.483339 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.483352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.486010 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.486019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.486207 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.486240 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.486341 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.486502 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":4,"thread_start_us":446,"threads_started":1}
I20260808 15:42:47.486999 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.474548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.23.130.2:34787)
0808 15:42:47.478827 (+ 4279us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.478833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.478856 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:42:47.478937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.478942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.479113 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.479312 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:47.479867 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.481336 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.483505 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.484971 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.485110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.485353 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.486559 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.486729 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4132,"thread_start_us":4019,"threads_started":1}
I20260808 15:42:47.490468 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e (DEFAULT_TABLE table=test-table [id=e48d73e429bd4b158dceae7f7243ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:47.490465 32110 tablet_service.cc:1467] Processing CreateTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e (DEFAULT_TABLE table=test-table [id=e48d73e429bd4b158dceae7f7243ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:47.491825 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769f2bb7a90f4438b1aaea648f6c239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:47.491855 32110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769f2bb7a90f4438b1aaea648f6c239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:47.492271 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.476610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34879 (local address 127.0.0.1:48342)
0808 15:42:47.480777 (+ 4167us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.480808 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.481001 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.482054 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.482057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.482079 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:47.484930 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.484945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.487170 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.487175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.489266 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.489305 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.491471 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.491479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.491665 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.491705 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.491809 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.491951 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3952,"thread_start_us":3759,"threads_started":1}
I20260808 15:42:47.492647 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.476965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.23.130.3:34879)
0808 15:42:47.477924 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.477936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.480906 (+ 2970us) server_negotiation.cc:407] Connection header received
0808 15:42:47.481227 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.481235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.481469 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.481664 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:47.485183 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.486985 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.489490 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.490962 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.491099 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.491304 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.491969 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.492143 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":470,"threads_started":1}
I20260808 15:42:47.496265 32266 tablet_service.cc:1467] Processing CreateTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e (DEFAULT_TABLE table=test-table [id=e48d73e429bd4b158dceae7f7243ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:47.497710 32266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769f2bb7a90f4438b1aaea648f6c239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:47.504966 32375 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Bootstrap starting.
I20260808 15:42:47.507421 32376 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Bootstrap starting.
I20260808 15:42:47.508810 32375 tablet_bootstrap.cc:654] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:47.510267 32375 log.cc:824] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:47.511077 32376 tablet_bootstrap.cc:654] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:47.512418 32376 log.cc:824] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:47.513017 32378 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Bootstrap starting.
I20260808 15:42:47.513922 32375 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: No bootstrap required, opened a new log
I20260808 15:42:47.514199 32375 ts_tablet_manager.cc:1397] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260808 15:42:47.517724 32378 tablet_bootstrap.cc:654] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:47.519065 32378 log.cc:824] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:47.519771 32376 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: No bootstrap required, opened a new log
I20260808 15:42:47.520040 32376 ts_tablet_manager.cc:1397] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260808 15:42:47.522697 32378 tablet_bootstrap.cc:492] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: No bootstrap required, opened a new log
I20260808 15:42:47.523092 32378 ts_tablet_manager.cc:1397] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 15:42:47.523854 32375 raft_consensus.cc:348] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.524298 32375 raft_consensus.cc:374] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:47.524399 32375 raft_consensus.cc:729] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ff6962684d8db69f0265bdd39310, State: Initialized, Role: FOLLOWER
I20260808 15:42:47.525004 32375 consensus_queue.cc:260] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.526758 32375 ts_tablet_manager.cc:1428] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260808 15:42:47.530649 32376 raft_consensus.cc:348] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.531086 32376 raft_consensus.cc:374] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:47.531201 32376 raft_consensus.cc:729] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c502541071445aa8c9805607dd731f, State: Initialized, Role: FOLLOWER
I20260808 15:42:47.531837 32376 consensus_queue.cc:260] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.534390 32376 ts_tablet_manager.cc:1428] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260808 15:42:47.534737 32180 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
I20260808 15:42:47.535256 32378 raft_consensus.cc:348] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.535683 32378 raft_consensus.cc:374] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:47.535784 32378 raft_consensus.cc:729] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0c4cb87e7547c5b08393456f710052, State: Initialized, Role: FOLLOWER
I20260808 15:42:47.536387 32378 consensus_queue.cc:260] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.539209 32336 heartbeater.cc:503] Master 127.23.130.62:46165 was elected leader, sending a full tablet report...
I20260808 15:42:47.539785 32378 ts_tablet_manager.cc:1428] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260808 15:42:47.606431 32382 raft_consensus.cc:482] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:47.606747 32382 raft_consensus.cc:504] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.608510 32382 leader_election.cc:288] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311), aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
W20260808 15:42:47.617022 32018 tablet.cc:2369] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:47.617810 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.609018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:41557 (local address 127.23.130.3:34879)
0808 15:42:47.609307 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.609317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.609893 (+ 576us) server_negotiation.cc:407] Connection header received
0808 15:42:47.610130 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.610136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.610220 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.610330 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:47.611151 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.612265 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.615472 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.616790 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.616943 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.617070 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.617417 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.617547 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:42:47.618026 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34879 (local address 127.23.130.2:41557)
0808 15:42:47.609758 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.609783 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.609947 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.610383 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.610387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.610456 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:47.610968 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.610987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.612417 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.612423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.615294 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.615311 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.617006 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.617014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.617229 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.617265 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.617375 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.617519 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":198,"spinlock_wait_cycles":6528,"thread_start_us":608,"threads_started":1}
I20260808 15:42:47.618088 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.609035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:52581 (local address 127.23.130.1:40311)
0808 15:42:47.609559 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.609568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.610056 (+ 488us) server_negotiation.cc:407] Connection header received
0808 15:42:47.610191 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.610196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.610273 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.610360 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:47.611152 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.612265 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.615484 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.616732 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.616976 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.617055 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.617693 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.617826 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260808 15:42:47.618795 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40311 (local address 127.23.130.2:52581)
0808 15:42:47.609939 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:47.609961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:47.610074 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:47.610380 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:47.610386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:47.610448 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:47.610987 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.610997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.612418 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.612423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.615294 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:47.615311 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.617303 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.617309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:47.617499 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.617532 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:42:47.618240 (+ 708us) client_negotiation.cc:240] Negotiation successful
0808 15:42:47.618420 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":238,"spinlock_wait_cycles":768,"thread_start_us":877,"threads_started":1}
I20260808 15:42:47.619148 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769f2bb7a90f4438b1aaea648f6c239e" candidate_uuid: "24c502541071445aa8c9805607dd731f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0c4cb87e7547c5b08393456f710052" is_pre_election: true
I20260808 15:42:47.619788 32287 raft_consensus.cc:2454] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c502541071445aa8c9805607dd731f in term 0.
I20260808 15:42:47.619807 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769f2bb7a90f4438b1aaea648f6c239e" candidate_uuid: "24c502541071445aa8c9805607dd731f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1681ff6962684d8db69f0265bdd39310" is_pre_election: true
I20260808 15:42:47.620330 31963 raft_consensus.cc:2454] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c502541071445aa8c9805607dd731f in term 0.
I20260808 15:42:47.620527 32055 leader_election.cc:302] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c502541071445aa8c9805607dd731f, aa0c4cb87e7547c5b08393456f710052; no voters:
I20260808 15:42:47.621007 32382 raft_consensus.cc:2781] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:47.621168 32382 raft_consensus.cc:482] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:47.621279 32382 raft_consensus.cc:3037] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:47.623771 32382 raft_consensus.cc:504] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
I20260808 15:42:47.624370 32382 leader_election.cc:288] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [CANDIDATE]: Term 1 election: Requested vote from peers 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311), aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:47.624732 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769f2bb7a90f4438b1aaea648f6c239e" candidate_uuid: "24c502541071445aa8c9805607dd731f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1681ff6962684d8db69f0265bdd39310"
I20260808 15:42:47.624954 31963 raft_consensus.cc:3037] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:47.624908 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769f2bb7a90f4438b1aaea648f6c239e" candidate_uuid: "24c502541071445aa8c9805607dd731f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0c4cb87e7547c5b08393456f710052"
I20260808 15:42:47.625128 32287 raft_consensus.cc:3037] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:47.628289 31963 raft_consensus.cc:2454] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c502541071445aa8c9805607dd731f in term 1.
I20260808 15:42:47.628548 32287 raft_consensus.cc:2454] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c502541071445aa8c9805607dd731f in term 1.
I20260808 15:42:47.628811 32057 leader_election.cc:302] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ff6962684d8db69f0265bdd39310, 24c502541071445aa8c9805607dd731f; no voters:
I20260808 15:42:47.629119 32382 raft_consensus.cc:2781] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:47.629596 32382 raft_consensus.cc:686] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 1 LEADER]: Becoming Leader. State: Replica: 24c502541071445aa8c9805607dd731f, State: Running, Role: LEADER
I20260808 15:42:47.630203 32382 consensus_queue.cc:237] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } }
W20260808 15:42:47.635705 32338 tablet.cc:2369] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:47.635823 30935 catalog_manager.cc:5495] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c502541071445aa8c9805607dd731f (127.23.130.2). New cstate: current_term: 1 leader_uuid: "24c502541071445aa8c9805607dd731f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:47.675038 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.670106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.23.130.2:34787)
0808 15:42:47.670289 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.670294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.670313 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:47.670437 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.670444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.670542 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.670673 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:47.671010 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.671732 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.673037 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.673427 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.673527 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.673932 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:47.674541 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:47.674665 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.674757 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.674827 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:42:47.683259 32287 raft_consensus.cc:1261] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Refusing update from remote peer 24c502541071445aa8c9805607dd731f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:47.683382 31963 raft_consensus.cc:1261] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Refusing update from remote peer 24c502541071445aa8c9805607dd731f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:47.684298 32388 consensus_queue.cc:1035] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:47.684705 32382 consensus_queue.cc:1035] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:47.697274 32392 mvcc.cc:205] Tried to move back new op lower bound from 7316290632420716544 to 7316290632217329664. Current Snapshot: MvccSnapshot[applied={T|T < 7316290632420716544}]
I20260808 15:42:47.716472 30935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:47.716747 30935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:47.724581 30935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c35d5c8533344b349a964b9002cf0401: Sending DeleteTablet for 3 replicas of tablet 769f2bb7a90f4438b1aaea648f6c239e
I20260808 15:42:47.725637 32110 tablet_service.cc:1514] Processing DeleteTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:47 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:36170
I20260808 15:42:47.725734 31943 tablet_service.cc:1514] Processing DeleteTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:47 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:47434
I20260808 15:42:47.726069 32266 tablet_service.cc:1514] Processing DeleteTablet for tablet 769f2bb7a90f4438b1aaea648f6c239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:47 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:48342
I20260808 15:42:47.728228 32402 tablet_replica.cc:330] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: stopping tablet replica
I20260808 15:42:47.728689 32402 raft_consensus.cc:2229] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:47.730109 32402 raft_consensus.cc:2258] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:47.734982 32404 tablet_replica.cc:330] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: stopping tablet replica
I20260808 15:42:47.735477 32404 raft_consensus.cc:2229] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:47.736037 32404 raft_consensus.cc:2258] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:47.736682 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 30902
I20260808 15:42:47.738458 30967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:47.739343 32402 ts_tablet_manager.cc:1905] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:47.745261 32404 ts_tablet_manager.cc:1905] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:47.747251 32402 ts_tablet_manager.cc:1918] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:47.747504 32402 log.cc:1196] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/769f2bb7a90f4438b1aaea648f6c239e
I20260808 15:42:47.748147 32402 ts_tablet_manager.cc:1939] T 769f2bb7a90f4438b1aaea648f6c239e P 24c502541071445aa8c9805607dd731f: Deleting consensus metadata
I20260808 15:42:47.750451 32403 tablet_replica.cc:330] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: stopping tablet replica
I20260808 15:42:47.750934 32403 raft_consensus.cc:2229] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:47.751595 32403 raft_consensus.cc:2258] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:47.753804 32403 ts_tablet_manager.cc:1905] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:47.757288 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.729090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.23.130.61:44743)
0808 15:42:47.729327 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.729336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.735107 (+ 5771us) server_negotiation.cc:407] Connection header received
0808 15:42:47.735403 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.735411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.735498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.735582 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.738270 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.739696 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.742275 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.743212 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.743341 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.748291 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.748302 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.748310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.748365 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.752518 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.754059 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.754068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.754074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.754279 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.755183 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.755192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.755197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.755674 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.756019 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.756684 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.756915 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:42:47.761112 32404 ts_tablet_manager.cc:1918] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:47.761294 32404 log.cc:1196] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/769f2bb7a90f4438b1aaea648f6c239e
I20260808 15:42:47.761885 32404 ts_tablet_manager.cc:1939] T 769f2bb7a90f4438b1aaea648f6c239e P aa0c4cb87e7547c5b08393456f710052: Deleting consensus metadata
I20260808 15:42:47.771632 32403 ts_tablet_manager.cc:1918] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:47.771808 32403 log.cc:1196] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/769f2bb7a90f4438b1aaea648f6c239e
I20260808 15:42:47.772358 32403 ts_tablet_manager.cc:1939] T 769f2bb7a90f4438b1aaea648f6c239e P 1681ff6962684d8db69f0265bdd39310: Deleting consensus metadata
I20260808 15:42:47.780875 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.735255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.23.130.60:46375)
0808 15:42:47.735508 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.735518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.759697 (+ 24179us) server_negotiation.cc:407] Connection header received
0808 15:42:47.759840 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.759847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.759948 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.760051 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.765862 (+ 5811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.767204 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.769304 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.769962 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.770062 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.773869 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.773876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.773882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.773925 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.777604 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.778603 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.778609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.778613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.778777 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.779238 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.779244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.779248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.779617 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.779876 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.780285 (+ 409us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.780527 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:47.900476 30923 catalog_manager.cc:4841] TS 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311): tablet 769f2bb7a90f4438b1aaea648f6c239e (table test-table [id=e48d73e429bd4b158dceae7f7243ad61]) successfully deleted
I20260808 15:42:47.901240 30921 catalog_manager.cc:4841] TS 24c502541071445aa8c9805607dd731f (127.23.130.2:34787): tablet 769f2bb7a90f4438b1aaea648f6c239e (table test-table [id=e48d73e429bd4b158dceae7f7243ad61]) successfully deleted
W20260808 15:42:47.901458 30921 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.130.2:34787, user_credentials={real_user=slave}} blocked reactor thread for 151537us
I20260808 15:42:47.901880 30921 catalog_manager.cc:4841] TS aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879): tablet 769f2bb7a90f4438b1aaea648f6c239e (table test-table [id=e48d73e429bd4b158dceae7f7243ad61]) successfully deleted
I20260808 15:42:47.902174 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 30902
I20260808 15:42:47.961210 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.940160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.23.130.60:46375)
0808 15:42:47.940358 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.940365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.943999 (+ 3634us) server_negotiation.cc:407] Connection header received
0808 15:42:47.944244 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.944250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.944382 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.944488 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.946905 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.947971 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.950058 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.950777 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.950891 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.954159 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.954166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.954171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.954213 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.958096 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.958842 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.958850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.958856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.958998 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.959443 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.959450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.959454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.959808 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.960060 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.960595 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.960841 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:42:47.963543 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:47.940151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.23.130.61:44743)
0808 15:42:47.940363 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:47.940368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:47.945677 (+ 5309us) server_negotiation.cc:407] Connection header received
0808 15:42:47.946473 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:47.946482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:47.946593 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:47.946682 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:47.949730 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.951077 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.952998 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:47.953771 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:47.953914 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:47.957589 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:47.957595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:47.957601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:47.957642 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:47.960624 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.961293 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.961298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.961301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.961593 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:47.962033 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:47.962040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:47.962044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:47.962347 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:47.962651 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:47.962983 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 15:42:47.963219 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:42:48.703697 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:48.702641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:57085)
0808 15:42:48.703274 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:48.703440 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":356,"threads_started":1}
W20260808 15:42:48.704146 32336 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:48.709767 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:48.708986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:47727)
0808 15:42:48.709438 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:48.709568 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
W20260808 15:42:48.710394 32014 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:48.712122 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:48.711497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:38131)
0808 15:42:48.711856 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:48.711976 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":179,"threads_started":1}
W20260808 15:42:48.712464 32180 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:49.224110 32419 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c35d5c8533344b349a964b9002cf0401)
I20260808 15:42:49.224453 32419 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } }
I20260808 15:42:49.224891 32420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c35d5c8533344b349a964b9002cf0401)
I20260808 15:42:49.225098 32420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } }
I20260808 15:42:49.232074 32420 leader_election.cc:288] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165), e30490cb4fa5493db178256bded15f28 (127.23.130.61:44743)
I20260808 15:42:49.232364 32419 leader_election.cc:288] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165), 6740ffbd9db54408a62339daaa5773de (127.23.130.60:46375)
I20260808 15:42:49.252400 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45308)
0808 15:42:49.251900 (+ 25854us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.252124 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25721,"threads_started":1}
I20260808 15:42:49.254271 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45314)
0808 15:42:49.253659 (+ 27593us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.253942 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27412,"threads_started":1}
W20260808 15:42:49.255834 31798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
W20260808 15:42:49.256620 31585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:49.267478 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.266722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45326)
0808 15:42:49.267037 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.267191 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260808 15:42:49.267714 31798 leader_election.cc:334] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165): Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:49.269708 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:44743 (local address 127.0.0.1:44322)
0808 15:42:49.252659 (+ 26596us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.252683 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.252893 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.253748 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.253756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.253794 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.254133 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.254149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.257778 (+ 3629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.257787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.261843 (+ 4056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.261864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.268872 (+ 7008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.268878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.269065 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.269101 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.269205 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.269380 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26484,"thread_start_us":25557,"threads_started":1}
I20260808 15:42:49.270058 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.23.130.61:44743)
0808 15:42:49.252304 (+ 25922us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.252312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.253068 (+ 756us) server_negotiation.cc:407] Connection header received
0808 15:42:49.253177 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.253186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.253328 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.253452 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.254382 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.257531 (+ 3149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.264351 (+ 6820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.268517 (+ 4166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.268640 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.268767 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.269257 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.269435 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25779,"thread_start_us":25877,"threads_started":1}
I20260808 15:42:49.270615 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.269745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45328)
0808 15:42:49.270012 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.270165 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260808 15:42:49.270875 31611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6740ffbd9db54408a62339daaa5773de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e30490cb4fa5493db178256bded15f28" is_pre_election: true
W20260808 15:42:49.270903 31585 leader_election.cc:334] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165): Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:49.271318 31611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6740ffbd9db54408a62339daaa5773de in term 1.
I20260808 15:42:49.271903 31800 leader_election.cc:302] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6740ffbd9db54408a62339daaa5773de, e30490cb4fa5493db178256bded15f28; no voters: c35d5c8533344b349a964b9002cf0401
I20260808 15:42:49.272154 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:46375 (local address 127.0.0.1:37566)
0808 15:42:49.258709 (+ 32599us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.258735 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.258889 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.262095 (+ 3206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.262103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.262145 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.262551 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.262566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.266925 (+ 4359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.266933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.269372 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.269387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.271374 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.271380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.271579 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.271635 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.271738 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.271886 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32467,"mutex_wait_us":1,"thread_start_us":28552,"threads_started":1}
I20260808 15:42:49.272318 32420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 15:42:49.272444 32420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Starting leader election (detected failure of leader c35d5c8533344b349a964b9002cf0401)
I20260808 15:42:49.272495 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.226605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.23.130.60:46375)
0808 15:42:49.253053 (+ 26448us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.253063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.259033 (+ 5970us) server_negotiation.cc:407] Connection header received
0808 15:42:49.259116 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.259124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.259232 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.259339 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.262795 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.266757 (+ 3962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.269531 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.271002 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.271141 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.271252 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.271780 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.271927 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26256,"thread_start_us":26028,"threads_started":1}
I20260808 15:42:49.272526 32420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:42:49.273100 31824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30490cb4fa5493db178256bded15f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6740ffbd9db54408a62339daaa5773de" is_pre_election: true
I20260808 15:42:49.275266 32420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } }
I20260808 15:42:49.275895 31824 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e30490cb4fa5493db178256bded15f28 in current term 2: Already voted for candidate 6740ffbd9db54408a62339daaa5773de in this term.
I20260808 15:42:49.276104 32420 leader_election.cc:288] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 election: Requested vote from peers c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165), e30490cb4fa5493db178256bded15f28 (127.23.130.61:44743)
I20260808 15:42:49.276600 31611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6740ffbd9db54408a62339daaa5773de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e30490cb4fa5493db178256bded15f28"
I20260808 15:42:49.276742 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.276213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45344)
0808 15:42:49.276441 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.276536 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:42:49.276814 31611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:42:49.277269 31584 leader_election.cc:302] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30490cb4fa5493db178256bded15f28; no voters: 6740ffbd9db54408a62339daaa5773de, c35d5c8533344b349a964b9002cf0401
I20260808 15:42:49.278017 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.277504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45360)
0808 15:42:49.277804 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.277888 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 15:42:49.278223 31798 leader_election.cc:334] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165): Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
I20260808 15:42:49.279457 31611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6740ffbd9db54408a62339daaa5773de in term 2.
I20260808 15:42:49.279639 32419 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:42:49.280056 31800 leader_election.cc:302] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6740ffbd9db54408a62339daaa5773de, e30490cb4fa5493db178256bded15f28; no voters: c35d5c8533344b349a964b9002cf0401
I20260808 15:42:49.280371 32420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:42:49.280995 32420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [term 2 LEADER]: Becoming Leader. State: Replica: 6740ffbd9db54408a62339daaa5773de, State: Running, Role: LEADER
I20260808 15:42:49.281378 32420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } }
I20260808 15:42:49.283489 32431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6740ffbd9db54408a62339daaa5773de. Latest consensus state: current_term: 2 leader_uuid: "6740ffbd9db54408a62339daaa5773de" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:49.283916 32431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:49.284636 32433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:49.288388 32433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:49.289222 32433 catalog_manager.cc:1259] Loaded cluster ID: 68cf3bdc3afd467b817b4137a427bcf7
I20260808 15:42:49.289336 32433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:49.290634 32433 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:49.291487 32433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Loaded TSK: 0
I20260808 15:42:49.292312 32433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:42:49.318230 31813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:49.325861 31813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:49.330148 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.329668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45374)
0808 15:42:49.329866 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.329963 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:42:49.330148 31611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [term 2 FOLLOWER]: Refusing update from remote peer 6740ffbd9db54408a62339daaa5773de: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 15:42:49.330932 32431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 15:42:49.331240 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.330812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45384)
0808 15:42:49.330998 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.331090 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 15:42:49.331418 31798 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de -> Peer c35d5c8533344b349a964b9002cf0401 (127.23.130.62:46165): Couldn't send request to peer c35d5c8533344b349a964b9002cf0401. Status: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 15:42:49.333684 32419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6740ffbd9db54408a62339daaa5773de. Latest consensus state: current_term: 2 leader_uuid: "6740ffbd9db54408a62339daaa5773de" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:49.333956 32419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:49.335012 32434 mvcc.cc:205] Tried to move back new op lower bound from 7316290639169634304 to 7316290638980759552. Current Snapshot: MvccSnapshot[applied={T|T < 7316290639169634304}]
I20260808 15:42:49.336174 32419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6740ffbd9db54408a62339daaa5773de" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:49.336530 32419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30490cb4fa5493db178256bded15f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:42:49.339385 32440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6740ffbd9db54408a62339daaa5773de" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c35d5c8533344b349a964b9002cf0401" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46165 } } peers { permanent_uuid: "e30490cb4fa5493db178256bded15f28" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 44743 } attrs { promote: false } } peers { permanent_uuid: "6740ffbd9db54408a62339daaa5773de" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 46375 } attrs { promote: false } } }
I20260808 15:42:49.339831 32440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:49.355341 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.347936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34879 (local address 127.0.0.1:48350)
0808 15:42:49.348061 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.348079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.348191 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.349205 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.349213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.349245 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.349634 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.349644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.351212 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.351220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.352666 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.352688 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.354240 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.354246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.354551 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.354591 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.354694 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.354888 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":3456}
I20260808 15:42:49.356083 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.348327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.23.130.3:34879)
0808 15:42:49.348810 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.348817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.348837 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:42:49.348915 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.348923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.349037 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.349166 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.350092 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.351068 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.352847 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.354022 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.354162 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.354293 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.355580 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.355740 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20260808 15:42:49.356904 32266 tablet_service.cc:1467] Processing CreateTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 (DEFAULT_TABLE table=test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:49.357431 32266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391fc06fe364f1fa313f84046fb3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:49.361150 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.347415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34787 (local address 127.0.0.1:36190)
0808 15:42:49.347642 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.347664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.347811 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.355062 (+ 7251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.355070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.355113 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.355380 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.355391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.357062 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.357070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.358806 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.358818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.360418 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.360423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.360604 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.360639 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.360700 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.360824 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3}
I20260808 15:42:49.361896 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.347517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.23.130.2:34787)
0808 15:42:49.348571 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.348578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.348600 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:42:49.354790 (+ 6190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.354798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.354898 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.355021 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.355646 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.356734 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.358971 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.360231 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.361421 (+ 1190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.361527 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.361578 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.361689 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":807,"threads_started":1}
I20260808 15:42:49.362725 32110 tablet_service.cc:1467] Processing CreateTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 (DEFAULT_TABLE table=test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:49.363282 32110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391fc06fe364f1fa313f84046fb3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:49.363576 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.348330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40311 (local address 127.0.0.1:47440)
0808 15:42:49.349160 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.349178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.349299 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.355900 (+ 6601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.355905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.355928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.356245 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.356256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.357726 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.357731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.360969 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.360982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.362749 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.362757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.362951 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.362982 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.363083 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.363221 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":1261,"thread_start_us":543,"threads_started":1}
I20260808 15:42:49.363686 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.348754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.23.130.1:40311)
0808 15:42:49.355564 (+ 6810us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.355571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.355592 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:42:49.355665 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.355670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.355749 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.355873 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.356406 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.357548 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.361147 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.362539 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.362694 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.362827 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.363134 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.363309 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6669,"thread_start_us":6544,"threads_started":1}
I20260808 15:42:49.364543 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 (DEFAULT_TABLE table=test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:49.365058 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391fc06fe364f1fa313f84046fb3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:49.368078 32445 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Bootstrap starting.
I20260808 15:42:49.370645 32445 tablet_bootstrap.cc:654] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:49.370968 32446 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Bootstrap starting.
I20260808 15:42:49.372462 32449 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Bootstrap starting.
I20260808 15:42:49.373117 32445 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: No bootstrap required, opened a new log
I20260808 15:42:49.373236 32445 ts_tablet_manager.cc:1397] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260808 15:42:49.373916 32445 raft_consensus.cc:348] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.374110 32445 raft_consensus.cc:374] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:49.374186 32445 raft_consensus.cc:729] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0c4cb87e7547c5b08393456f710052, State: Initialized, Role: FOLLOWER
I20260808 15:42:49.374230 32449 tablet_bootstrap.cc:654] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:49.374486 32445 consensus_queue.cc:260] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.376959 32446 tablet_bootstrap.cc:654] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:49.383816 32449 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: No bootstrap required, opened a new log
I20260808 15:42:49.383976 32449 ts_tablet_manager.cc:1397] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260808 15:42:49.385885 32449 raft_consensus.cc:348] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.386140 32449 raft_consensus.cc:374] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:49.386250 32449 raft_consensus.cc:729] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ff6962684d8db69f0265bdd39310, State: Initialized, Role: FOLLOWER
I20260808 15:42:49.386354 32446 tablet_bootstrap.cc:492] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: No bootstrap required, opened a new log
I20260808 15:42:49.386606 32446 ts_tablet_manager.cc:1397] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260808 15:42:49.386946 32449 consensus_queue.cc:260] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.387313 32455 raft_consensus.cc:482] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:49.387573 32455 raft_consensus.cc:504] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.387719 32446 raft_consensus.cc:348] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.387984 32446 raft_consensus.cc:374] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:49.388109 32446 raft_consensus.cc:729] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c502541071445aa8c9805607dd731f, State: Initialized, Role: FOLLOWER
I20260808 15:42:49.388222 32445 ts_tablet_manager.cc:1428] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Time spent starting tablet: real 0.015s user 0.003s sys 0.012s
I20260808 15:42:49.388417 32446 consensus_queue.cc:260] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.390787 32446 ts_tablet_manager.cc:1428] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260808 15:42:49.390826 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.389691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:47971)
0808 15:42:49.390358 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.390546 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":325,"threads_started":1}
I20260808 15:42:49.393913 32335 heartbeater.cc:503] Master 127.23.130.60:46375 was elected leader, sending a full tablet report...
I20260808 15:42:49.394335 32179 heartbeater.cc:503] Master 127.23.130.60:46375 was elected leader, sending a full tablet report...
I20260808 15:42:49.400663 32449 ts_tablet_manager.cc:1428] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Time spent starting tablet: real 0.015s user 0.006s sys 0.005s
I20260808 15:42:49.400815 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.391446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:47179)
0808 15:42:49.400419 (+ 8973us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.400547 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8859,"threads_started":1}
I20260808 15:42:49.401995 32455 leader_election.cc:288] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24c502541071445aa8c9805607dd731f (127.23.130.2:34787), aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:49.403685 32012 heartbeater.cc:503] Master 127.23.130.60:46375 was elected leader, sending a full tablet report...
I20260808 15:42:49.410128 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.403762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:57699)
0808 15:42:49.409555 (+ 5793us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.409822 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5653,"thread_start_us":5538,"threads_started":1}
I20260808 15:42:49.412551 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.392056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:42639)
0808 15:42:49.412109 (+ 20053us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.412259 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19896,"threads_started":1}
I20260808 15:42:49.412601 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.400673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34787 (local address 127.23.130.1:55277)
0808 15:42:49.402590 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.402609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.402797 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.403357 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.403361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.403386 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.403783 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.403796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.405706 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.405714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.408907 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.408928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.411613 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.411623 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.411919 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.411961 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.412067 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.412217 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"thread_start_us":2692,"threads_started":1}
I20260808 15:42:49.413182 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.401141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:55277 (local address 127.23.130.2:34787)
0808 15:42:49.401362 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.401371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.402688 (+ 1317us) server_negotiation.cc:407] Connection header received
0808 15:42:49.402979 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.402984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.403077 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.403140 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.404083 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.405472 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.409102 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.410999 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.411265 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.411384 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.412778 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.412937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:42:49.413806 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.413411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:38633)
0808 15:42:49.413559 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.413662 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 15:42:49.415277 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391fc06fe364f1fa313f84046fb3e35" candidate_uuid: "1681ff6962684d8db69f0265bdd39310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c502541071445aa8c9805607dd731f" is_pre_election: true
I20260808 15:42:49.415671 32130 raft_consensus.cc:2454] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1681ff6962684d8db69f0265bdd39310 in term 0.
I20260808 15:42:49.416324 31890 leader_election.cc:302] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ff6962684d8db69f0265bdd39310, 24c502541071445aa8c9805607dd731f; no voters:
I20260808 15:42:49.416604 32463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.414980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:40979)
0808 15:42:49.416112 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.416301 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1000}
I20260808 15:42:49.416929 32456 raft_consensus.cc:2781] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:49.417098 32456 raft_consensus.cc:482] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:49.417300 32456 raft_consensus.cc:3037] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:49.418087 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.410084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49819 (local address 127.23.130.3:34879)
0808 15:42:49.410257 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.410262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.410609 (+ 347us) server_negotiation.cc:407] Connection header received
0808 15:42:49.410913 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.410922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.411020 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.411093 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:49.412020 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.413413 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.415818 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.416908 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.417094 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.417186 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.417824 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.417921 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:42:49.418322 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.409824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34879 (local address 127.23.130.1:49819)
0808 15:42:49.410495 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.410517 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:49.410721 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:49.411328 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:49.411336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:49.411368 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:49.411821 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.411836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.413656 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.413662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.415645 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:49.415664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.417374 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.417384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:49.417617 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.417661 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 15:42:49.417791 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 15:42:49.417974 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":4443,"threads_started":1}
I20260808 15:42:49.419106 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391fc06fe364f1fa313f84046fb3e35" candidate_uuid: "1681ff6962684d8db69f0265bdd39310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0c4cb87e7547c5b08393456f710052" is_pre_election: true
I20260808 15:42:49.419363 32287 raft_consensus.cc:2454] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1681ff6962684d8db69f0265bdd39310 in term 0.
I20260808 15:42:49.420645 32456 raft_consensus.cc:504] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.421329 32456 leader_election.cc:288] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [CANDIDATE]: Term 1 election: Requested vote from peers 24c502541071445aa8c9805607dd731f (127.23.130.2:34787), aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879)
I20260808 15:42:49.421761 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391fc06fe364f1fa313f84046fb3e35" candidate_uuid: "1681ff6962684d8db69f0265bdd39310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c502541071445aa8c9805607dd731f"
I20260808 15:42:49.421983 32130 raft_consensus.cc:3037] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:49.421919 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391fc06fe364f1fa313f84046fb3e35" candidate_uuid: "1681ff6962684d8db69f0265bdd39310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0c4cb87e7547c5b08393456f710052"
I20260808 15:42:49.422183 32287 raft_consensus.cc:3037] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:49.424606 32130 raft_consensus.cc:2454] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1681ff6962684d8db69f0265bdd39310 in term 1.
I20260808 15:42:49.424736 32287 raft_consensus.cc:2454] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1681ff6962684d8db69f0265bdd39310 in term 1.
I20260808 15:42:49.425452 31890 leader_election.cc:302] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ff6962684d8db69f0265bdd39310, aa0c4cb87e7547c5b08393456f710052; no voters:
I20260808 15:42:49.425750 32456 raft_consensus.cc:2781] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:49.425954 32456 raft_consensus.cc:686] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 1 LEADER]: Becoming Leader. State: Replica: 1681ff6962684d8db69f0265bdd39310, State: Running, Role: LEADER
I20260808 15:42:49.426394 32456 consensus_queue.cc:237] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } }
I20260808 15:42:49.426959 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.426540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:33853)
0808 15:42:49.426704 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.426811 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:42:49.430264 31811 catalog_manager.cc:5495] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1681ff6962684d8db69f0265bdd39310 (127.23.130.1). New cstate: current_term: 1 leader_uuid: "1681ff6962684d8db69f0265bdd39310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1681ff6962684d8db69f0265bdd39310" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40311 } health_report { overall_health: HEALTHY } } }
I20260808 15:42:49.456569 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.451042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.23.130.1:40311)
0808 15:42:49.451293 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.451300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.451319 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:49.451481 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.451485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.451571 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.451729 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:49.452167 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.452912 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.454285 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.454813 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.454951 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.455393 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:49.455995 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:49.456150 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.456209 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.456291 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:42:49.460490 32287 raft_consensus.cc:1261] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Refusing update from remote peer 1681ff6962684d8db69f0265bdd39310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:49.460937 32130 raft_consensus.cc:1261] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Refusing update from remote peer 1681ff6962684d8db69f0265bdd39310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:49.461280 32455 consensus_queue.cc:1035] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0c4cb87e7547c5b08393456f710052" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:49.461700 32456 consensus_queue.cc:1035] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c502541071445aa8c9805607dd731f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:49.464778 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.463860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:41047)
0808 15:42:49.464154 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.464472 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 15:42:49.465590 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.465093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:45449)
0808 15:42:49.465256 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.465348 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:42:49.469725 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.469036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.3:52551)
0808 15:42:49.469379 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.469495 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260808 15:42:49.473201 32466 mvcc.cc:205] Tried to move back new op lower bound from 7316290639703629824 to 7316290639575281664. Current Snapshot: MvccSnapshot[applied={T|T < 7316290639703629824}]
I20260808 15:42:49.475395 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.474972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.2:47949)
0808 15:42:49.475125 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.475236 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:42:49.480973 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.480414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:45247)
0808 15:42:49.480665 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.480791 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 15:42:49.483903 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.483498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.23.130.1:42409)
0808 15:42:49.483701 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.483773 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 15:42:49.493109 31813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:49.493357 31813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:49.498747 31813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6740ffbd9db54408a62339daaa5773de: Sending DeleteTablet for 3 replicas of tablet 4391fc06fe364f1fa313f84046fb3e35
I20260808 15:42:49.499761 32110 tablet_service.cc:1514] Processing DeleteTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:49 UTC) from {username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:36190
I20260808 15:42:49.499853 32266 tablet_service.cc:1514] Processing DeleteTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:49 UTC) from {username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:48350
I20260808 15:42:49.500113 31943 tablet_service.cc:1514] Processing DeleteTablet for tablet 4391fc06fe364f1fa313f84046fb3e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:49 UTC) from {username='oryx', principal='oryx/127.23.130.60@KRBTEST.COM'} at 127.0.0.1:47440
I20260808 15:42:49.501729 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31851
I20260808 15:42:49.511399 32479 tablet_replica.cc:330] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: stopping tablet replica
I20260808 15:42:49.511693 32479 raft_consensus.cc:2229] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:49.512058 32479 raft_consensus.cc:2258] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:49.514566 32481 tablet_replica.cc:330] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: stopping tablet replica
I20260808 15:42:49.514818 32481 raft_consensus.cc:2229] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:49.515640 32481 raft_consensus.cc:2258] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:49.519096 32481 ts_tablet_manager.cc:1905] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:49.519691 32479 ts_tablet_manager.cc:1905] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:49.521628 32476 tablet_replica.cc:330] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: stopping tablet replica
I20260808 15:42:49.521840 32476 raft_consensus.cc:2229] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:49.522219 32476 raft_consensus.cc:2258] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:49.527410 32476 ts_tablet_manager.cc:1905] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:49.528420 32481 ts_tablet_manager.cc:1918] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:49.528644 32481 log.cc:1196] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/4391fc06fe364f1fa313f84046fb3e35
I20260808 15:42:49.529300 32481 ts_tablet_manager.cc:1939] T 4391fc06fe364f1fa313f84046fb3e35 P 1681ff6962684d8db69f0265bdd39310: Deleting consensus metadata
I20260808 15:42:49.531461 31800 catalog_manager.cc:4841] TS 1681ff6962684d8db69f0265bdd39310 (127.23.130.1:40311): tablet 4391fc06fe364f1fa313f84046fb3e35 (table test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]) successfully deleted
I20260808 15:42:49.534027 32476 ts_tablet_manager.cc:1918] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:49.534180 32476 log.cc:1196] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/4391fc06fe364f1fa313f84046fb3e35
I20260808 15:42:49.534720 32476 ts_tablet_manager.cc:1939] T 4391fc06fe364f1fa313f84046fb3e35 P 24c502541071445aa8c9805607dd731f: Deleting consensus metadata
I20260808 15:42:49.535856 31798 catalog_manager.cc:4841] TS 24c502541071445aa8c9805607dd731f (127.23.130.2:34787): tablet 4391fc06fe364f1fa313f84046fb3e35 (table test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]) successfully deleted
I20260808 15:42:49.536367 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.502770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.23.130.1:40311)
0808 15:42:49.502988 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.502997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.516719 (+ 13722us) server_negotiation.cc:407] Connection header received
0808 15:42:49.517022 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.517030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.517145 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.517306 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:49.521237 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.522487 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.524628 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.525289 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.525399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.529315 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:49.529335 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:49.529361 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:49.529406 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:49.532894 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.533621 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.533630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.533634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.533807 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.534238 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.534245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.534251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.534791 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:49.535108 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.535730 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.535949 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:42:49.537474 32006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:49.537557 32479 ts_tablet_manager.cc:1918] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:49.537707 32479 log.cc:1196] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/4391fc06fe364f1fa313f84046fb3e35
I20260808 15:42:49.538280 32479 ts_tablet_manager.cc:1939] T 4391fc06fe364f1fa313f84046fb3e35 P aa0c4cb87e7547c5b08393456f710052: Deleting consensus metadata
I20260808 15:42:49.539708 31798 catalog_manager.cc:4841] TS aa0c4cb87e7547c5b08393456f710052 (127.23.130.3:34879): tablet 4391fc06fe364f1fa313f84046fb3e35 (table test-table [id=7027f32dd45a4fc9a01c4000c251c3c9]) successfully deleted
I20260808 15:42:49.641417 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31851
I20260808 15:42:49.659672 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 32028
I20260808 15:42:49.672775 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.660465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.23.130.2:34787)
0808 15:42:49.660650 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.660654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.660672 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:42:49.660807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.660812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.660873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.660978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:49.662208 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.662992 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.664215 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.664599 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.664691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.667339 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:49.667366 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:49.667382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:49.667416 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:49.670297 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.671033 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.671040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.671042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.671173 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.671493 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.671496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.671498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.671837 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:49.672081 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.672327 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.672518 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:42:49.673503 32173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:49.805392 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.804819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45392)
0808 15:42:49.805026 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.805168 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:42:49.805974 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 32028
I20260808 15:42:49.806918 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.806169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46165 (local address 127.0.0.1:45406)
0808 15:42:49.806705 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:49.806795 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430}
I20260808 15:42:49.832191 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 32194
I20260808 15:42:49.845773 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:49.833153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.23.130.3:34879)
0808 15:42:49.833345 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:49.833350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:49.833369 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:49.833546 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:49.833550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:49.833621 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:49.833724 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:49.835263 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.836112 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.837245 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:49.837665 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:49.837759 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:49.840428 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:49.840443 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:49.840459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:49.840493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:49.843235 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.843873 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.843881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.843883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.844010 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:49.844346 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:49.844350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:49.844352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:49.844712 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:49.844946 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:49.845195 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 15:42:49.845389 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:42:49.846470 32330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:49.941845 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 32194
I20260808 15:42:49.962697 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31443
I20260808 15:42:49.963850 31634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:50.084280 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31443
I20260808 15:42:50.108040 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 31561
I20260808 15:42:50.109154 31846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:50.236586 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 31561
2026-08-08T15:42:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9810 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:50 dist-test-slave-gp6n krb5kdc[32490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:50 dist-test-slave-gp6n krb5kdc[32490](info): set up 2 sockets
Aug 08 15:42:50 dist-test-slave-gp6n krb5kdc[32490](info): commencing operation
krb5kdc: starting...
W20260808 15:42:51.421176 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:51 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab.
Entry for principal oryx/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab.
I20260808 15:42:51.683135 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42979
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab
--principal=oryx/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:42979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:52.096252 32503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:52.096621 32503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:52.096794 32503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:52.103950 32503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:52.104095 32503 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:42:52.104202 32503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:52.104327 32503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:52.104423 32503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:52.104522 32503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:52.114236 32503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:42979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.62.keytab
--principal=oryx/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:52.115653 32503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:52.117025 32503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:52.128999 32508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:52.129123 32510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:52.129155 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:52.130239 32503 server_base.cc:1034] running on GCE node
I20260808 15:42:52.131182 32503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:52.132757 32503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:52.134059 32503 hybrid_clock.cc:648] HybridClock initialized: now 1786203772133999 us; error 47 us; skew 500 ppm
I20260808 15:42:52.134577 32503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:52 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203772, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:52.138090 32503 init.cc:377] Logged in from keytab as oryx/127.23.130.62@KRBTEST.COM (short username oryx)
I20260808 15:42:52.139698 32503 webserver.cc:460] Webserver started at http://127.23.130.62:39889/ using document root <none> and password file <none>
I20260808 15:42:52.140473 32503 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:52.140618 32503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:52.140980 32503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:52.144310 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "e94344fa4a9b4461b9035c467a560d6c"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "82163e8739dad1949ed0a95a02fd8f74"
server_key_iv: "4dae4c623bfde413cf25c57563d4bda0"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.145215 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "e94344fa4a9b4461b9035c467a560d6c"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "82163e8739dad1949ed0a95a02fd8f74"
server_key_iv: "4dae4c623bfde413cf25c57563d4bda0"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.150889 32503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:42:52.154690 32520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:52.156219 32503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:52.156464 32503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "e94344fa4a9b4461b9035c467a560d6c"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "82163e8739dad1949ed0a95a02fd8f74"
server_key_iv: "4dae4c623bfde413cf25c57563d4bda0"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.156713 32503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:52.172160 32503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:52.176992 32503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:52.177309 32503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:52.200517 32503 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:42979
I20260808 15:42:52.200539 32591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:42979 every 8 connection(s)
I20260808 15:42:52.202124 32503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:42:52.206462 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:52.211266 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 32503
I20260808 15:42:52.211539 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:42:52.212213 24072 external_mini_cluster.cc:1428] Setting key a83c14ad13f0fbbeb4fa837028d7a55e
I20260808 15:42:52.219786 32592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Bootstrap starting.
I20260808 15:42:52.224624 32592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:52.226084 32592 log.cc:824] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:52.230048 32592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: No bootstrap required, opened a new log
Aug 08 15:42:52 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:52.241881 32592 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } }
I20260808 15:42:52.242380 32592 raft_consensus.cc:374] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:52.242509 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:52.214197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.23.130.62:42979)
0808 15:42:52.215165 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:52.215183 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:52.215241 (+ 58us) server_negotiation.cc:407] Connection header received
0808 15:42:52.218042 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:52.218073 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:52.218696 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:52.219158 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:52.221449 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.223841 (+ 2392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.225918 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.227350 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.227584 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:52.232828 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:52.232839 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:52.232865 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:52.232915 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:52.237493 (+ 4578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.238243 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.238253 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.238263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.238527 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.238976 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.238983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.238987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.239839 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:52.240164 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:52.240518 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 15:42:52.241025 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":458,"threads_started":1}
I20260808 15:42:52.242548 32592 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94344fa4a9b4461b9035c467a560d6c, State: Initialized, Role: FOLLOWER
I20260808 15:42:52.243199 32592 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } }
I20260808 15:42:52.243499 32592 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:42:52.243669 32592 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:42:52.243885 32592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:52.246905 32592 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } }
I20260808 15:42:52.247419 32592 leader_election.cc:302] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94344fa4a9b4461b9035c467a560d6c; no voters:
I20260808 15:42:52.248015 32592 leader_election.cc:288] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:42:52.248174 32597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:52.248927 32597 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [term 1 LEADER]: Becoming Leader. State: Replica: e94344fa4a9b4461b9035c467a560d6c, State: Running, Role: LEADER
I20260808 15:42:52.249565 32597 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } }
I20260808 15:42:52.250126 32592 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:42:52.253376 32599 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94344fa4a9b4461b9035c467a560d6c. Latest consensus state: current_term: 1 leader_uuid: "e94344fa4a9b4461b9035c467a560d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } } }
I20260808 15:42:52.253455 32598 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94344fa4a9b4461b9035c467a560d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94344fa4a9b4461b9035c467a560d6c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42979 } } }
I20260808 15:42:52.253777 32599 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:52.253849 32598 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c [sys.catalog]: This master's current role is: LEADER
I20260808 15:42:52.260433 32604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:42:52.265702 32604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:42:52.274524 32604 catalog_manager.cc:1347] Generated new cluster ID: 9762b3a06d3247ea88a0996cff287303
I20260808 15:42:52.274637 32604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:42:52.315008 32604 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:42:52.317147 32604 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:42:52.330363 32604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Generated new TSK 0
I20260808 15:42:52.331509 32604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
I20260808 15:42:52.398943 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42979
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:52.741467 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:52.741752 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:52.741945 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:52.749341 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:52.749454 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:52.749634 32621 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:52.749747 32621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:42:52.760058 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:52.761545 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:52.763062 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:52.776178 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:52.776238 32633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:52.776424 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:52.777488 32621 server_base.cc:1034] running on GCE node
I20260808 15:42:52.778079 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:52.779058 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:52.780303 32621 hybrid_clock.cc:648] HybridClock initialized: now 1786203772780225 us; error 57 us; skew 500 ppm
I20260808 15:42:52.780611 32621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:52 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203772, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:52.783973 32621 init.cc:377] Logged in from keytab as oryx/127.23.130.1@KRBTEST.COM (short username oryx)
I20260808 15:42:52.785471 32621 webserver.cc:460] Webserver started at http://127.23.130.1:34397/ using document root <none> and password file <none>
I20260808 15:42:52.786176 32621 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:52.786265 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:52.786628 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:52.789505 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "babfe465183826341a44ef75c8cdf183"
server_key_iv: "71f83ac90d6c6a595f20b24dafd572cd"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.790241 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "babfe465183826341a44ef75c8cdf183"
server_key_iv: "71f83ac90d6c6a595f20b24dafd572cd"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.795404 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:42:52.799019 32640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:52.800491 32621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:42:52.800637 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
format_stamp: "Formatted at 2026-08-08 15:42:52 on dist-test-slave-gp6n"
server_key: "babfe465183826341a44ef75c8cdf183"
server_key_iv: "71f83ac90d6c6a595f20b24dafd572cd"
server_key_version: "encryptionkey@0"
I20260808 15:42:52.800840 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:52.825017 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:52.829909 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:52.830226 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:52.833088 32621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:52.833225 32621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:52.833457 32621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:52.833585 32621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 15:42:52 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203772, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:52.858357 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:52.833716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:56597 (local address 127.23.130.62:42979)
0808 15:42:52.834651 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:52.834660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:52.835636 (+ 976us) server_negotiation.cc:407] Connection header received
0808 15:42:52.837651 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:52.837656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:52.837749 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:52.837865 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:52.839979 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.840772 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.843369 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.844278 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.844457 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:52.850124 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:52.850134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:52.850141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:52.850188 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:52.853761 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.855108 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.855117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.855124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.855310 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.855937 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.855943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.855946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.856351 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:52.856521 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:52.857419 (+ 898us) server_negotiation.cc:299] Negotiation successful
0808 15:42:52.857766 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":668,"threads_started":1}
I20260808 15:42:52.858796 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:52.834502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.1:56597)
0808 15:42:52.835439 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:52.835491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:52.837359 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:52.838104 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:52.838112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:52.838922 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:52.839785 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:52.839807 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.841070 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.841077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:52.843164 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:52.843182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.844609 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.844618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:52.844949 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:52.845858 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:52.845885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:52.849774 (+ 3889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:52.853939 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:52.853949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:52.853973 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:52.854888 (+ 915us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:52.855457 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:52.855462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:52.855467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:52.855805 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:52.856592 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:52.856597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:52.856954 (+ 357us) client_negotiation.cc:769] Sending connection context
0808 15:42:52.857184 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 15:42:52.857570 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":405,"threads_started":1}
I20260808 15:42:52.879714 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:52.864794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:47275 (local address 127.23.130.62:42979)
0808 15:42:52.865027 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:52.865037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:52.865601 (+ 564us) server_negotiation.cc:407] Connection header received
0808 15:42:52.865743 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:52.865753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:52.865889 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:52.866032 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:52.867166 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.868536 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.869877 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.870886 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.871052 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:52.872723 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:52.872734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:52.872742 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:52.872793 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:52.876295 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.877169 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.877177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.877184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.877356 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:52.877879 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:52.877882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:52.877884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:52.878161 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:52.878283 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:52.879020 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 15:42:52.879293 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:42:52.880157 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:52.864793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.1:47275)
0808 15:42:52.865496 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:52.865521 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:52.865624 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:52.866038 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:52.866041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:52.866538 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:52.866988 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:52.867006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.868692 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.868696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:52.869720 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:52.869731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:52.871078 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:52.871083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:52.871200 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:52.871800 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:52.871816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:52.872503 (+ 687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:52.876427 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:52.876432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:52.876440 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:52.877039 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:52.877495 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:52.877508 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:52.877515 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:52.877733 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:52.878297 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:52.878305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:52.878640 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 15:42:52.878891 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 15:42:52.879407 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":394,"threads_started":1}
I20260808 15:42:52.892765 32621 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:37303
I20260808 15:42:52.892788 306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:37303 every 8 connection(s)
I20260808 15:42:52.894075 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:42:52.896126 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 32621
I20260808 15:42:52.896346 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:42:52.896643 307 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42979
I20260808 15:42:52.896896 307 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:52.897042 24072 external_mini_cluster.cc:1428] Setting key 9095ce4f32120c1e306ec55fe2e7dba9
I20260808 15:42:52.897590 307 heartbeater.cc:511] Master 127.23.130.62:42979 requested a full tablet report, sending...
I20260808 15:42:52.899755 32544 ts_manager.cc:194] Registered new tserver with Master: ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303)
I20260808 15:42:52.901213 32544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:56597
WARNING: no policy specified for oryx/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
I20260808 15:42:52.952046 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42979
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:53.294147 313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:53.294516 313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:53.294699 313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:53.301937 313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:53.302038 313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:53.302204 313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:53.302342 313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:42:53.312315 313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:53.313531 313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:53.314954 313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:53.328286 320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:53.328480 319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:53.328331 322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:53.329149 313 server_base.cc:1034] running on GCE node
I20260808 15:42:53.329757 313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:53.330879 313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:53.332119 313 hybrid_clock.cc:648] HybridClock initialized: now 1786203773332047 us; error 50 us; skew 500 ppm
I20260808 15:42:53.332412 313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:53 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203773, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:53.335572 313 init.cc:377] Logged in from keytab as oryx/127.23.130.2@KRBTEST.COM (short username oryx)
I20260808 15:42:53.337013 313 webserver.cc:460] Webserver started at http://127.23.130.2:41305/ using document root <none> and password file <none>
I20260808 15:42:53.337695 313 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:53.337826 313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:53.338153 313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:53.341082 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "34e9987be79f4c46b36fb5b42f64aae2"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "bf6edcc6f8b8b49ce0c31a627abdafa3"
server_key_iv: "ea9dc67b99258bc5c8a19c0f2114ca27"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.341925 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "34e9987be79f4c46b36fb5b42f64aae2"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "bf6edcc6f8b8b49ce0c31a627abdafa3"
server_key_iv: "ea9dc67b99258bc5c8a19c0f2114ca27"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.347064 313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:42:53.350354 330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:53.351856 313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:42:53.352013 313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "34e9987be79f4c46b36fb5b42f64aae2"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "bf6edcc6f8b8b49ce0c31a627abdafa3"
server_key_iv: "ea9dc67b99258bc5c8a19c0f2114ca27"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.352226 313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:53.371731 313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:53.376091 313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:53.376389 313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:53.379396 313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:53.379524 313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:53.379734 313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:53.379827 313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:42:53 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203773, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:53.409714 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.380258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.2:49473)
0808 15:42:53.381286 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:53.381353 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:53.384498 (+ 3145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:53.385376 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:53.385388 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:53.386637 (+ 1249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:53.387920 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.387961 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.389382 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.389391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.392705 (+ 3314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.392727 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.394417 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.394427 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.394720 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.395529 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:53.395576 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:53.399345 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:53.403869 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.403879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.403899 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.404647 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.405150 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.405157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.405164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.405520 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.406235 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:53.406245 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:53.406732 (+ 487us) client_negotiation.cc:769] Sending connection context
0808 15:42:53.407158 (+ 426us) client_negotiation.cc:240] Negotiation successful
0808 15:42:53.407807 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":498,"threads_started":1}
I20260808 15:42:53.411128 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.379662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:49473 (local address 127.23.130.62:42979)
0808 15:42:53.379905 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:53.379914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:53.381480 (+ 1566us) server_negotiation.cc:407] Connection header received
0808 15:42:53.384862 (+ 3382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:53.384867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:53.384969 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:53.385094 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:53.388200 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.389199 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.392960 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.393912 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.394062 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.399631 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:53.399637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:53.399641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:53.399678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:53.403697 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.404840 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.404846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.404850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.404995 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.405656 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.405665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.405669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.406089 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:53.406280 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:53.410521 (+ 4241us) server_negotiation.cc:299] Negotiation successful
0808 15:42:53.410707 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:53.432482 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.417425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:44555 (local address 127.23.130.62:42979)
0808 15:42:53.417678 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:53.417687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:53.418326 (+ 639us) server_negotiation.cc:407] Connection header received
0808 15:42:53.418527 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:53.418534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:53.418641 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:53.418794 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:53.419704 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.421132 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.422718 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.423566 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.423741 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.426116 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:53.426122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:53.426127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:53.426167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:53.429518 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.430470 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.430476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.430479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.430615 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.430989 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.430996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.430999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.431326 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:53.431493 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:53.431888 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 15:42:53.432110 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:42:53.432627 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.417442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.2:44555)
0808 15:42:53.418220 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:53.418237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:53.418352 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:53.418823 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:53.418827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:53.419257 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:53.419543 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.419552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.421294 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.421301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.422557 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.422570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.423799 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.423808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.424046 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.424751 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:53.424781 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:53.425909 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:53.429664 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.429669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.429673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.430304 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.430732 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.430736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.430740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.430873 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.431448 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:53.431452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:53.431651 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 15:42:53.431793 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 15:42:53.432153 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":430,"threads_started":1}
I20260808 15:42:53.435632 313 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41073
I20260808 15:42:53.435662 464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41073 every 8 connection(s)
I20260808 15:42:53.437004 313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:42:53.438359 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 313
I20260808 15:42:53.438870 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:42:53.439255 465 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42979
I20260808 15:42:53.439503 465 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:53.439524 24072 external_mini_cluster.cc:1428] Setting key 9544f6ecd2929eb6cae9304850978589
I20260808 15:42:53.440268 465 heartbeater.cc:511] Master 127.23.130.62:42979 requested a full tablet report, sending...
I20260808 15:42:53.441812 32544 ts_manager.cc:194] Registered new tserver with Master: 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073)
I20260808 15:42:53.442993 32544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:49473
WARNING: no policy specified for oryx/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
I20260808 15:42:53.501737 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42979
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:53.836459 470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:53.836786 470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:53.836967 470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:53.844198 470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:53.844360 470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:53.844521 470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:42:53.844667 470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:42:53.854970 470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.130.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:53.856341 470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:53.857718 470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:53.871870 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:53.871937 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:53.872031 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:53.872843 470 server_base.cc:1034] running on GCE node
I20260808 15:42:53.873492 470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:53.874517 470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:53.875757 470 hybrid_clock.cc:648] HybridClock initialized: now 1786203773875681 us; error 55 us; skew 500 ppm
I20260808 15:42:53.876071 470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:42:53 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203773, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:42:53.879400 470 init.cc:377] Logged in from keytab as oryx/127.23.130.3@KRBTEST.COM (short username oryx)
I20260808 15:42:53.880910 470 webserver.cc:460] Webserver started at http://127.23.130.3:46199/ using document root <none> and password file <none>
I20260808 15:42:53.881639 470 fs_manager.cc:359] Metadata directory not provided
I20260808 15:42:53.881772 470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:42:53.882135 470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:42:53.885241 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "040054f4cc4f455aad09b77ee3322e54"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "b7650a099abba791599fded962a2c692"
server_key_iv: "e360b6834377138e020647dcf3521795"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.886068 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "040054f4cc4f455aad09b77ee3322e54"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "b7650a099abba791599fded962a2c692"
server_key_iv: "e360b6834377138e020647dcf3521795"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.891522 470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:42:53.896014 488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:53.897643 470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:42:53.897814 470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "040054f4cc4f455aad09b77ee3322e54"
format_stamp: "Formatted at 2026-08-08 15:42:53 on dist-test-slave-gp6n"
server_key: "b7650a099abba791599fded962a2c692"
server_key_iv: "e360b6834377138e020647dcf3521795"
server_key_version: "encryptionkey@0"
I20260808 15:42:53.898056 470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:42:53.904563 307 heartbeater.cc:503] Master 127.23.130.62:42979 was elected leader, sending a full tablet report...
I20260808 15:42:53.911720 470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:42:53.915900 470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:42:53.916179 470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:42:53.918893 470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:42:53.919003 470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:42:53.919171 470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:42:53.919251 470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:42:53 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203773, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:53.943251 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.919622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:34199 (local address 127.23.130.62:42979)
0808 15:42:53.919823 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:53.919829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:53.921365 (+ 1536us) server_negotiation.cc:407] Connection header received
0808 15:42:53.923666 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:53.923675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:53.923792 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:53.923925 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:53.926500 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.928001 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.929679 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.930580 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.930724 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.935274 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:53.935281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:53.935287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:53.935338 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:53.938776 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.939856 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.939860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.939862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.940032 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.940628 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.940633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.940637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.941071 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:53.941244 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:53.942644 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0808 15:42:53.942885 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:42:53.944123 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.920268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.3:34199)
0808 15:42:53.921166 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:53.921215 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:53.923257 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:53.924218 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:53.924230 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:53.925012 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:53.926260 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.926296 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.928193 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.928197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.929524 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.929536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.930846 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.930852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.931034 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.931859 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:53.931884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:53.934998 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:53.938979 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.938990 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.939015 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.939679 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.940196 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.940203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.940210 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.940496 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.941214 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:53.941226 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:53.941752 (+ 526us) client_negotiation.cc:769] Sending connection context
0808 15:42:53.942108 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 15:42:53.942788 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":403,"threads_started":1}
I20260808 15:42:53.965135 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.949924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:52965 (local address 127.23.130.62:42979)
0808 15:42:53.950104 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:53.950109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:53.951052 (+ 943us) server_negotiation.cc:407] Connection header received
0808 15:42:53.951388 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:53.951394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:53.951478 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:53.951549 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:53.952835 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.953857 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.955836 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.956384 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.956538 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.958432 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:53.958439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:53.958443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:53.958486 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:53.962150 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.963087 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.963095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.963098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.963240 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:53.963623 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:53.963628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:53.963632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:53.963920 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:53.964109 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:53.964514 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 15:42:53.964744 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 15:42:53.965209 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:53.949938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.23.130.3:52965)
0808 15:42:53.950939 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:53.950962 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:53.951211 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:53.951755 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:53.951759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:53.952315 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:53.952688 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.952698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.954004 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.954010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.955691 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:53.955710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:53.956581 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:53.956587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:53.956796 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:53.957475 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:53.957492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:53.958213 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:53.962303 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.962309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.962318 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.962940 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.963355 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:53.963360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:53.963364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:53.963505 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:53.964069 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:53.964074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:53.964265 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 15:42:53.964404 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 15:42:53.964753 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":497,"threads_started":1}
I20260808 15:42:53.980079 470 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:39629
I20260808 15:42:53.980103 619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:39629 every 8 connection(s)
I20260808 15:42:53.981444 470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:42:53.983669 620 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42979
I20260808 15:42:53.983870 620 heartbeater.cc:464] Registering TS with master...
I20260808 15:42:53.984510 620 heartbeater.cc:511] Master 127.23.130.62:42979 requested a full tablet report, sending...
I20260808 15:42:53.985883 32544 ts_manager.cc:194] Registered new tserver with Master: 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:53.986819 32544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:34199
I20260808 15:42:53.987682 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 470
I20260808 15:42:53.987907 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:42:53.988353 24072 external_mini_cluster.cc:1428] Setting key 9d4f2023b0918dbb73b5f4f34888ecb8
I20260808 15:42:53.991397 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:42:54 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:42:54 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:42:54 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:42:54.446130 465 heartbeater.cc:503] Master 127.23.130.62:42979 was elected leader, sending a full tablet report...
I20260808 15:42:54.989660 620 heartbeater.cc:503] Master 127.23.130.62:42979 was elected leader, sending a full tablet report...
W20260808 15:42:55.269886 629 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.224s user 0.000s sys 0.019s
W20260808 15:42:55.271329 32651 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:54.039806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.23.130.62:42979)
0808 15:42:54.040053 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:54.040059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:54.040238 (+ 179us) server_negotiation.cc:407] Connection header received
0808 15:42:54.040519 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:54.040527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:54.040604 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:54.040716 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:54.042153 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:54.042979 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:54.044283 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:54.044708 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:54.044867 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.270921 (+1226054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54020: BlockingRecv error: recv got EOF from 127.0.0.1:54020 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 08 15:42:55 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:55.290793 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.275635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.23.130.62:42979)
0808 15:42:55.275815 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.275820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.276270 (+ 450us) server_negotiation.cc:407] Connection header received
0808 15:42:55.276395 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.276399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.276506 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.276604 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:55.277837 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.278716 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.279998 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.280439 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.280539 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.284460 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:55.284465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:55.284469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:55.284506 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:55.288085 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:55.288765 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:55.288771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:55.288774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:55.288954 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:55.289474 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:55.289479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:55.289482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:55.289802 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:55.290038 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.290310 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.290564 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:42:55.295161 32544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:55.302129 32544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:55.342813 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.317892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:37303 (local address 127.0.0.1:40742)
0808 15:42:55.333774 (+ 15882us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.333791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.334068 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.335118 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.335124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.335159 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.335784 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.335798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.337178 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.337184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.339427 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.339443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.341571 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.341578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.341792 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.341851 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.342004 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.342218 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15772,"mutex_wait_us":1,"spinlock_wait_cycles":2944,"thread_start_us":13992,"threads_started":1}
I20260808 15:42:55.343370 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.314882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.23.130.3:39629)
0808 15:42:55.334685 (+ 19803us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.334694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.334724 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:42:55.334856 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.334865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.335109 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.335333 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.336144 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.337843 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.340573 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.341628 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.341763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.341982 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.342923 (+ 941us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.343090 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19625,"thread_start_us":19466,"threads_started":1}
I20260808 15:42:55.343261 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.314681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.0.0.1:34316)
0808 15:42:55.329509 (+ 14828us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.329563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.329803 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.335329 (+ 5526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.335334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.335371 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.335949 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.335960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.338032 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.338036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.340422 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.340434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.342136 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.342143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.342311 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.342344 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.342471 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.342617 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14500,"mutex_wait_us":41,"spinlock_wait_cycles":1280,"thread_start_us":15329,"threads_started":1}
I20260808 15:42:55.343761 647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.318150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.23.130.1:37303)
0808 15:42:55.333357 (+ 15207us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.333365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.334015 (+ 650us) server_negotiation.cc:407] Connection header received
0808 15:42:55.334511 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.334519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.334726 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.334958 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.336028 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.337043 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.339634 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.341347 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.341523 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.341770 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.342063 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.342242 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15050,"thread_start_us":14922,"threads_started":1}
I20260808 15:42:55.344807 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.314965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41073 (local address 127.0.0.1:46800)
0808 15:42:55.332224 (+ 17259us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.332249 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.332413 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.334852 (+ 2439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.334860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.334891 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.335707 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.335734 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.338030 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.338036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.341173 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.341191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.343978 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.343985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.344164 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.344195 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.344278 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.344448 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17086,"mutex_wait_us":34,"spinlock_wait_cycles":1280,"thread_start_us":18564,"threads_started":1}
I20260808 15:42:55.344895 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.315279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.23.130.2:41073)
0808 15:42:55.332949 (+ 17670us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.332965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.332999 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:42:55.333125 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.333135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.333478 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.334335 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.336001 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.337838 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.341351 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.343339 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.343509 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.343730 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.344400 (+ 670us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.344572 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17518,"thread_start_us":17375,"threads_started":1}
I20260808 15:42:55.347795 32701 tablet_service.cc:1467] Processing CreateTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 (DEFAULT_TABLE table=test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.347795 543 tablet_service.cc:1467] Processing CreateTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 (DEFAULT_TABLE table=test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.349327 543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d0fb4bd444ab889f60f3f305bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.349329 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d0fb4bd444ab889f60f3f305bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.354449 388 tablet_service.cc:1467] Processing CreateTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 (DEFAULT_TABLE table=test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.355898 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d0fb4bd444ab889f60f3f305bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.365031 649 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Bootstrap starting.
I20260808 15:42:55.365887 648 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Bootstrap starting.
I20260808 15:42:55.370524 649 tablet_bootstrap.cc:654] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.371253 650 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Bootstrap starting.
I20260808 15:42:55.372161 649 log.cc:824] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:55.373174 648 tablet_bootstrap.cc:654] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.374526 648 log.cc:824] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:55.374833 650 tablet_bootstrap.cc:654] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.375389 649 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: No bootstrap required, opened a new log
I20260808 15:42:55.375679 649 ts_tablet_manager.cc:1397] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:42:55.376293 650 log.cc:824] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:42:55.379655 648 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: No bootstrap required, opened a new log
I20260808 15:42:55.379840 650 tablet_bootstrap.cc:492] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: No bootstrap required, opened a new log
I20260808 15:42:55.379942 648 ts_tablet_manager.cc:1397] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260808 15:42:55.380131 650 ts_tablet_manager.cc:1397] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:42:55.384680 649 raft_consensus.cc:348] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.385082 649 raft_consensus.cc:374] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.385243 649 raft_consensus.cc:729] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.385906 649 consensus_queue.cc:260] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.387638 649 ts_tablet_manager.cc:1428] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 15:42:55.390542 650 raft_consensus.cc:348] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.391003 650 raft_consensus.cc:374] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.391122 650 raft_consensus.cc:729] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.390890 648 raft_consensus.cc:348] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.391438 648 raft_consensus.cc:374] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.391556 648 raft_consensus.cc:729] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.391726 650 consensus_queue.cc:260] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.392206 648 consensus_queue.cc:260] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.393560 650 ts_tablet_manager.cc:1428] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 15:42:55.395085 648 ts_tablet_manager.cc:1428] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
W20260808 15:42:55.397806 308 tablet.cc:2369] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:42:55.440128 466 tablet.cc:2369] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:55.463912 658 raft_consensus.cc:482] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:55.464275 658 raft_consensus.cc:504] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.466053 658 leader_election.cc:288] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073), ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303)
I20260808 15:42:55.477739 647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.466632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:46983 (local address 127.23.130.1:37303)
0808 15:42:55.466852 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.466860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.467617 (+ 757us) server_negotiation.cc:407] Connection header received
0808 15:42:55.467893 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.467899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.467985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.468057 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.468714 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.469872 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.473007 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.474262 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.474393 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.474499 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.477321 (+ 2822us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.477473 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:42:55.477738 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.466615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:49277 (local address 127.23.130.2:41073)
0808 15:42:55.466854 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.466862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.467740 (+ 878us) server_negotiation.cc:407] Connection header received
0808 15:42:55.467922 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.467926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.467996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.468113 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.468905 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.469959 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.472999 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.474252 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.474393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.474554 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.477321 (+ 2767us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.477480 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:42:55.478551 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.466627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41073 (local address 127.23.130.3:49277)
0808 15:42:55.467626 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.467641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.467758 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.468149 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.468154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.468177 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.468709 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.468729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.470080 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.470089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.472801 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.472816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.477008 (+ 4192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.477014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.477152 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.477177 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.477949 (+ 772us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.478090 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":130,"thread_start_us":550,"threads_started":1}
I20260808 15:42:55.479027 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.466598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:37303 (local address 127.23.130.3:46983)
0808 15:42:55.467495 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.467511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.467713 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.468109 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.468114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.468155 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.468556 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.468569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.470010 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.470035 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.472807 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.472816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.476890 (+ 4074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.476898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.477099 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.477138 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.478312 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.478479 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":128,"thread_start_us":490,"threads_started":1}
I20260808 15:42:55.480346 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d0fb4bd444ab889f60f3f305bc33" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" is_pre_election: true
I20260808 15:42:55.480893 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d0fb4bd444ab889f60f3f305bc33" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2" is_pre_election: true
I20260808 15:42:55.481045 32727 raft_consensus.cc:2454] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 0.
I20260808 15:42:55.481552 408 raft_consensus.cc:2454] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 0.
I20260808 15:42:55.481845 491 leader_election.cc:302] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:55.482645 658 raft_consensus.cc:2781] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:55.482873 658 raft_consensus.cc:482] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:55.483016 658 raft_consensus.cc:3037] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Advancing to term 1
W20260808 15:42:55.484354 621 tablet.cc:2369] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:42:55.486102 658 raft_consensus.cc:504] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.487574 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d0fb4bd444ab889f60f3f305bc33" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2"
I20260808 15:42:55.487887 408 raft_consensus.cc:3037] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:55.488606 658 leader_election.cc:288] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 election: Requested vote from peers 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073), ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303)
I20260808 15:42:55.489220 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d0fb4bd444ab889f60f3f305bc33" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
I20260808 15:42:55.489494 32727 raft_consensus.cc:3037] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:55.491425 408 raft_consensus.cc:2454] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 1.
I20260808 15:42:55.491923 490 leader_election.cc:302] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, 34e9987be79f4c46b36fb5b42f64aae2; no voters:
I20260808 15:42:55.492267 658 raft_consensus.cc:2781] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:55.492844 32727 raft_consensus.cc:2454] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 1.
I20260808 15:42:55.492928 658 raft_consensus.cc:686] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 1 LEADER]: Becoming Leader. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Running, Role: LEADER
I20260808 15:42:55.493533 658 consensus_queue.cc:237] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } }
I20260808 15:42:55.498250 32543 catalog_manager.cc:5495] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:55.534633 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.529399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.23.130.3:39629)
0808 15:42:55.529655 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.529660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.529675 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:42:55.529759 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.529763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.529851 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.530014 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:55.530424 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.531158 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.532480 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.532896 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.533059 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.533475 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:55.534010 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:55.534170 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.534231 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.534313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:42:55.545035 32727 raft_consensus.cc:1261] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Refusing update from remote peer 040054f4cc4f455aad09b77ee3322e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:55.545215 408 raft_consensus.cc:1261] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Refusing update from remote peer 040054f4cc4f455aad09b77ee3322e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:55.546247 661 consensus_queue.cc:1035] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:55.546775 658 consensus_queue.cc:1035] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:55.580020 667 mvcc.cc:205] Tried to move back new op lower bound from 7316290664621957120 to 7316290664426205184. Current Snapshot: MvccSnapshot[applied={T|T < 7316290664621957120}]
I20260808 15:42:55.589550 32544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:55.589869 32544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:55.593681 32544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Sending DeleteTablet for 3 replicas of tablet d177d0fb4bd444ab889f60f3f305bc33
I20260808 15:42:55.594918 543 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:55 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:34316
I20260808 15:42:55.594990 388 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:55 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46800
I20260808 15:42:55.595667 32701 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d0fb4bd444ab889f60f3f305bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:55 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:40742
I20260808 15:42:55.598760 32544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:55.600349 32544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:55.612118 675 tablet_replica.cc:330] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: stopping tablet replica
I20260808 15:42:55.612943 542 tablet_service.cc:1467] Processing CreateTablet for tablet b0422692ac194623916d4504802a1da6 (DEFAULT_TABLE table=test-table [id=89b0f46251244d728baa477262bea691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.613493 677 tablet_replica.cc:330] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: stopping tablet replica
I20260808 15:42:55.613590 542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0422692ac194623916d4504802a1da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.613503 32701 tablet_service.cc:1467] Processing CreateTablet for tablet b0422692ac194623916d4504802a1da6 (DEFAULT_TABLE table=test-table [id=89b0f46251244d728baa477262bea691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.614017 677 raft_consensus.cc:2229] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:55.614606 677 raft_consensus.cc:2258] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:55.614826 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0422692ac194623916d4504802a1da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.615298 675 raft_consensus.cc:2229] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:55.615971 675 raft_consensus.cc:2258] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:55.618973 387 tablet_service.cc:1467] Processing CreateTablet for tablet b0422692ac194623916d4504802a1da6 (DEFAULT_TABLE table=test-table [id=89b0f46251244d728baa477262bea691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:55.619573 387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0422692ac194623916d4504802a1da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.624280 675 ts_tablet_manager.cc:1905] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:55.626293 677 ts_tablet_manager.cc:1905] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:55.626557 650 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Bootstrap starting.
I20260808 15:42:55.628291 650 tablet_bootstrap.cc:654] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.628999 648 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Bootstrap starting.
I20260808 15:42:55.631398 648 tablet_bootstrap.cc:654] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.636786 649 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Bootstrap starting.
I20260808 15:42:55.638626 649 tablet_bootstrap.cc:654] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.642647 648 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: No bootstrap required, opened a new log
I20260808 15:42:55.642930 648 ts_tablet_manager.cc:1397] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260808 15:42:55.643018 676 tablet_replica.cc:330] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: stopping tablet replica
I20260808 15:42:55.643707 676 raft_consensus.cc:2229] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:55.644352 676 raft_consensus.cc:2258] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:55.644222 648 raft_consensus.cc:348] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.646027 648 raft_consensus.cc:374] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.646234 648 raft_consensus.cc:729] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.647156 648 consensus_queue.cc:260] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.648196 650 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: No bootstrap required, opened a new log
I20260808 15:42:55.648221 676 ts_tablet_manager.cc:1905] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:55.648396 650 ts_tablet_manager.cc:1397] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260808 15:42:55.648752 649 tablet_bootstrap.cc:492] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: No bootstrap required, opened a new log
I20260808 15:42:55.648914 649 ts_tablet_manager.cc:1397] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260808 15:42:55.649210 648 ts_tablet_manager.cc:1428] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260808 15:42:55.649744 649 raft_consensus.cc:348] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.649839 650 raft_consensus.cc:348] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.649978 649 raft_consensus.cc:374] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.650077 649 raft_consensus.cc:729] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.650067 650 raft_consensus.cc:374] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.650269 650 raft_consensus.cc:729] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.650310 649 consensus_queue.cc:260] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.650647 650 consensus_queue.cc:260] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.651814 649 ts_tablet_manager.cc:1428] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:42:55.656512 650 ts_tablet_manager.cc:1428] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260808 15:42:55.659087 677 ts_tablet_manager.cc:1918] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:55.659274 677 log.cc:1196] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/d177d0fb4bd444ab889f60f3f305bc33
I20260808 15:42:55.660089 675 ts_tablet_manager.cc:1918] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:55.660291 675 log.cc:1196] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/d177d0fb4bd444ab889f60f3f305bc33
I20260808 15:42:55.660933 675 ts_tablet_manager.cc:1939] T d177d0fb4bd444ab889f60f3f305bc33 P 040054f4cc4f455aad09b77ee3322e54: Deleting consensus metadata
I20260808 15:42:55.662915 676 ts_tablet_manager.cc:1918] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:55.663084 676 log.cc:1196] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/d177d0fb4bd444ab889f60f3f305bc33
I20260808 15:42:55.663681 676 ts_tablet_manager.cc:1939] T d177d0fb4bd444ab889f60f3f305bc33 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting consensus metadata
I20260808 15:42:55.665620 32522 catalog_manager.cc:4841] TS 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073): tablet d177d0fb4bd444ab889f60f3f305bc33 (table test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]) successfully deleted
I20260808 15:42:55.666798 32525 catalog_manager.cc:4841] TS 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629): tablet d177d0fb4bd444ab889f60f3f305bc33 (table test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]) successfully deleted
I20260808 15:42:55.667009 677 ts_tablet_manager.cc:1939] T d177d0fb4bd444ab889f60f3f305bc33 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting consensus metadata
I20260808 15:42:55.668988 32523 catalog_manager.cc:4841] TS ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303): tablet d177d0fb4bd444ab889f60f3f305bc33 (table test-table [id=f73391cd40dd4a98b2ed6aeaff9409c1]) successfully deleted
I20260808 15:42:55.690553 656 raft_consensus.cc:482] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:55.690850 656 raft_consensus.cc:504] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.693046 656 leader_election.cc:288] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:55.714243 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.694001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49063 (local address 127.23.130.2:41073)
0808 15:42:55.694221 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.694229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.705058 (+ 10829us) server_negotiation.cc:407] Connection header received
0808 15:42:55.705657 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.705664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.705767 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.705838 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.707174 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.708428 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.711457 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.712890 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.713043 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.713134 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.713770 (+ 636us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.713928 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:42:55.715320 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.693445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41073 (local address 127.23.130.1:49063)
0808 15:42:55.704920 (+ 11475us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.704945 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.705400 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.706082 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.706087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.706123 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.706955 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.706972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.708647 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.708653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.711250 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.711272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.713364 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.713371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.713558 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.713594 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.714722 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.714888 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11312,"mutex_wait_us":38,"thread_start_us":11663,"threads_started":1}
I20260808 15:42:55.716630 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.694388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:42197 (local address 127.23.130.3:39629)
0808 15:42:55.694631 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.694639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.707192 (+ 12553us) server_negotiation.cc:407] Connection header received
0808 15:42:55.707648 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.707656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.707759 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.707827 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:55.708790 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.710163 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.713245 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.714947 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.715107 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.715209 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.716005 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.716175 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:42:55.717020 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0422692ac194623916d4504802a1da6" candidate_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2" is_pre_election: true
I20260808 15:42:55.717315 408 raft_consensus.cc:2454] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac418d7dab384f87ab44aaa8f0db0f0e in term 0.
I20260808 15:42:55.717935 32642 leader_election.cc:302] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e9987be79f4c46b36fb5b42f64aae2, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:55.718479 656 raft_consensus.cc:2781] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:55.718642 656 raft_consensus.cc:482] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:55.718752 656 raft_consensus.cc:3037] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:55.722604 656 raft_consensus.cc:504] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.723696 656 leader_election.cc:288] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [CANDIDATE]: Term 1 election: Requested vote from peers 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:55.723867 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.693567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.23.130.1:42197)
0808 15:42:55.707066 (+ 13499us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.707093 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.707466 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.708041 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.708048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.708079 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:55.708562 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.708575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.710363 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.710375 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.713052 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.713071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.715498 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.715507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.715770 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.715817 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.717653 (+ 1836us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.717834 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13339,"mutex_wait_us":3,"thread_start_us":13630,"threads_started":1}
I20260808 15:42:55.724053 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0422692ac194623916d4504802a1da6" candidate_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2"
I20260808 15:42:55.724285 408 raft_consensus.cc:3037] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:55.724779 569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0422692ac194623916d4504802a1da6" candidate_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54"
I20260808 15:42:55.725230 569 raft_consensus.cc:3037] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:55.725821 570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0422692ac194623916d4504802a1da6" candidate_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54" is_pre_election: true
I20260808 15:42:55.727694 408 raft_consensus.cc:2454] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac418d7dab384f87ab44aaa8f0db0f0e in term 1.
I20260808 15:42:55.728158 32642 leader_election.cc:302] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e9987be79f4c46b36fb5b42f64aae2, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:55.728479 656 raft_consensus.cc:2781] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:55.731163 569 raft_consensus.cc:2454] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac418d7dab384f87ab44aaa8f0db0f0e in term 1.
I20260808 15:42:55.732249 656 raft_consensus.cc:686] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 LEADER]: Becoming Leader. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Running, Role: LEADER
I20260808 15:42:55.732697 656 consensus_queue.cc:237] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.736449 32543 catalog_manager.cc:5495] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1). New cstate: current_term: 1 leader_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:42:55 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203772, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.62@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:55.840739 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.821085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.23.130.62:42979)
0808 15:42:55.821719 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:55.821725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:55.822290 (+ 565us) server_negotiation.cc:407] Connection header received
0808 15:42:55.822515 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:55.822523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:55.822619 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:55.822749 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:55.824334 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.825828 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.827769 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.828558 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.828705 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.833265 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:55.833271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:55.833277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:55.833314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:55.836955 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:55.837994 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:55.838001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:55.838007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:55.838159 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:55.838647 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:55.838653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:55.838656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:55.839064 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:55.839233 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:55.839800 (+ 567us) server_negotiation.cc:299] Negotiation successful
0808 15:42:55.840262 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":202,"server-negotiator.queue_time_us":480,"thread_start_us":338,"threads_started":1}
I20260808 15:42:55.840816 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:55.821055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42979 (local address 127.0.0.1:54040)
0808 15:42:55.822162 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:55.822183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:55.822353 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:55.822800 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:55.822805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:55.823779 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:42:55.824139 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.824151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.825990 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.825995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.827610 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:55.827626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:55.828772 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:55.828779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:55.828984 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:55.829836 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:42:55.829867 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:42:55.833040 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:42:55.837104 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:55.837121 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:55.837132 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:55.837821 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:55.838285 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:42:55.838292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:42:55.838298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:42:55.838504 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:42:55.839218 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:42:55.839230 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:42:55.839466 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 15:42:55.839665 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 15:42:55.840064 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":789,"threads_started":1}
I20260808 15:42:55.847759 32543 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:54040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:42:55.849182 32543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:55.857523 32701 tablet_service.cc:1467] Processing CreateTablet for tablet 19bf4bfc82f14a869a1bdd47fac28d24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea2778a3fc34f57a03728ad1e99778d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:55.857713 388 tablet_service.cc:1467] Processing CreateTablet for tablet 19bf4bfc82f14a869a1bdd47fac28d24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea2778a3fc34f57a03728ad1e99778d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:55.858197 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bf4bfc82f14a869a1bdd47fac28d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.858286 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bf4bfc82f14a869a1bdd47fac28d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.861655 542 tablet_service.cc:1467] Processing CreateTablet for tablet 19bf4bfc82f14a869a1bdd47fac28d24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea2778a3fc34f57a03728ad1e99778d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:42:55.862322 542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bf4bfc82f14a869a1bdd47fac28d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:55.865871 650 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: Bootstrap starting.
I20260808 15:42:55.867619 650 tablet_bootstrap.cc:654] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.878263 650 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: No bootstrap required, opened a new log
I20260808 15:42:55.878561 650 ts_tablet_manager.cc:1397] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:42:55.879402 650 raft_consensus.cc:348] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.879761 650 raft_consensus.cc:374] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.879894 650 raft_consensus.cc:729] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.880193 650 consensus_queue.cc:260] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.880717 657 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:55.881129 657 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:55.881664 650 ts_tablet_manager.cc:1428] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:42:55.886300 649 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: Bootstrap starting.
I20260808 15:42:55.888211 649 tablet_bootstrap.cc:654] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.888885 648 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: Bootstrap starting.
I20260808 15:42:55.891299 648 tablet_bootstrap.cc:654] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:55.899856 649 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: No bootstrap required, opened a new log
I20260808 15:42:55.900118 649 ts_tablet_manager.cc:1397] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260808 15:42:55.901019 649 raft_consensus.cc:348] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.901365 649 raft_consensus.cc:374] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.901492 649 raft_consensus.cc:729] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.901837 649 consensus_queue.cc:260] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.902370 656 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:55.903246 656 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:55.903831 649 ts_tablet_manager.cc:1428] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260808 15:42:55.909294 648 tablet_bootstrap.cc:492] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: No bootstrap required, opened a new log
I20260808 15:42:55.909476 648 ts_tablet_manager.cc:1397] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20260808 15:42:55.910228 648 raft_consensus.cc:348] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.910501 648 raft_consensus.cc:374] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:55.910579 648 raft_consensus.cc:729] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Initialized, Role: FOLLOWER
I20260808 15:42:55.910837 648 consensus_queue.cc:260] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:55.911296 672 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:55.911702 672 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:55.913370 648 ts_tablet_manager.cc:1428] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:42:56.114575 672 raft_consensus.cc:482] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:56.114818 672 raft_consensus.cc:504] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.115621 672 leader_election.cc:288] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073)
I20260808 15:42:56.116248 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bf4bfc82f14a869a1bdd47fac28d24" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" is_pre_election: true
I20260808 15:42:56.116274 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bf4bfc82f14a869a1bdd47fac28d24" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2" is_pre_election: true
I20260808 15:42:56.116513 32727 raft_consensus.cc:2454] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 0.
I20260808 15:42:56.116580 408 raft_consensus.cc:2454] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 0.
I20260808 15:42:56.116982 491 leader_election.cc:302] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:56.117326 672 raft_consensus.cc:2781] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:56.117482 672 raft_consensus.cc:482] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:56.117607 672 raft_consensus.cc:3037] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.120464 672 raft_consensus.cc:504] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.121050 672 leader_election.cc:288] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 election: Requested vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073)
I20260808 15:42:56.121536 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bf4bfc82f14a869a1bdd47fac28d24" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e9987be79f4c46b36fb5b42f64aae2"
I20260808 15:42:56.121611 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bf4bfc82f14a869a1bdd47fac28d24" candidate_uuid: "040054f4cc4f455aad09b77ee3322e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
I20260808 15:42:56.121809 408 raft_consensus.cc:3037] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.121858 32727 raft_consensus.cc:3037] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.124063 408 raft_consensus.cc:2454] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 1.
I20260808 15:42:56.124514 490 leader_election.cc:302] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, 34e9987be79f4c46b36fb5b42f64aae2; no voters:
I20260808 15:42:56.124797 672 raft_consensus.cc:2781] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:56.124881 32727 raft_consensus.cc:2454] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040054f4cc4f455aad09b77ee3322e54 in term 1.
I20260808 15:42:56.125294 672 raft_consensus.cc:686] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [term 1 LEADER]: Becoming Leader. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Running, Role: LEADER
I20260808 15:42:56.125634 672 consensus_queue.cc:237] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.127352 695 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: TxnStatusTablet state changed. Reason: New leader 040054f4cc4f455aad09b77ee3322e54. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.127666 695 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:56.128360 697 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:42:56.128643 697 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:42:56.129042 32543 catalog_manager.cc:5495] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } health_report { overall_health: HEALTHY } } }
I20260808 15:42:56.135779 656 consensus_queue.cc:1035] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.143019 683 consensus_queue.cc:1035] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.190707 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.185077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.0.0.1:34330)
0808 15:42:56.185326 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:56.185343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:56.185463 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:56.185881 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:56.185884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:56.185893 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:56.186053 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.186060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.187586 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.187591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.188580 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.188591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.189540 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.189544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.189734 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.189798 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.190230 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.190246 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 15:42:56.190306 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 15:42:56.190496 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:42:56.190768 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.185208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.23.130.3:39629)
0808 15:42:56.185384 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.185391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.185413 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:42:56.185651 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.185657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.185746 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.185842 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:56.186191 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.187439 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.188726 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.189376 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.189509 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.189981 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.190121 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.190233 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.190358 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.190466 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 15:42:56.194774 32727 raft_consensus.cc:1261] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Refusing update from remote peer 040054f4cc4f455aad09b77ee3322e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:56.194803 408 raft_consensus.cc:1261] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Refusing update from remote peer 040054f4cc4f455aad09b77ee3322e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:56.195420 695 consensus_queue.cc:1035] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.195755 672 consensus_queue.cc:1035] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.199456 657 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: TxnStatusTablet state changed. Reason: New leader 040054f4cc4f455aad09b77ee3322e54. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.199752 657 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:56.205703 657 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.205938 657 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 34e9987be79f4c46b36fb5b42f64aae2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:56.212438 702 mvcc.cc:205] Tried to move back new op lower bound from 7316290667287138304 to 7316290667012935680. Current Snapshot: MvccSnapshot[applied={T|T < 7316290667287138304}]
I20260808 15:42:56.213568 656 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: TxnStatusTablet state changed. Reason: New leader 040054f4cc4f455aad09b77ee3322e54. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.214037 656 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:56.239043 695 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.239374 695 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:56.246104 656 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.246461 656 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P ac418d7dab384f87ab44aaa8f0db0f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:42:56.253717 695 tablet_replica.cc:439] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040054f4cc4f455aad09b77ee3322e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } } }
I20260808 15:42:56.254022 695 tablet_replica.cc:442] T 19bf4bfc82f14a869a1bdd47fac28d24 P 040054f4cc4f455aad09b77ee3322e54: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:42:56.269208 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.262792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.23.130.1:37303)
0808 15:42:56.263500 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.263508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.263530 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:42:56.263642 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.263646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.263743 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.263943 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:56.264449 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.265247 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.266875 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.267335 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.267489 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.267950 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.268528 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.268701 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.268791 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.268882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":409,"threads_started":1}
I20260808 15:42:56.285938 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.279772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.23.130.1:49213)
0808 15:42:56.280387 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:56.280402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:56.280557 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:56.280900 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:56.280904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:56.280915 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:56.281190 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.281200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.282733 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.282737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.283800 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.283811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.284819 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.284823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.284937 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.284987 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.285399 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.285407 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:42:56.285538 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 15:42:56.285661 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":257,"threads_started":1}
I20260808 15:42:56.285985 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.279950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49213 (local address 127.23.130.3:39629)
0808 15:42:56.280172 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.280179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.280496 (+ 317us) server_negotiation.cc:407] Connection header received
0808 15:42:56.280696 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.280705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.280775 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.280886 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:56.281370 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.282586 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.283939 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.284663 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.284793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.285148 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.285300 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.285402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.285578 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.285662 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:42:56.328940 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.321682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:40681 (local address 127.23.130.1:37303)
0808 15:42:56.321842 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.321848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.322674 (+ 826us) server_negotiation.cc:407] Connection header received
0808 15:42:56.322804 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.322810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.322898 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.322989 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:56.323560 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.324714 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.326955 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.327539 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.327676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.328180 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.328313 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.328413 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.328575 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.328668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:42:56.329643 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.321775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:37303 (local address 127.23.130.3:40681)
0808 15:42:56.322580 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:56.322597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:56.322710 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:56.323018 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:56.323021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:56.323031 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:56.323371 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.323380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.324864 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.324869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.326800 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.326821 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.327735 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.327742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.327923 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.327998 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.328433 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.328444 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:42:56.328529 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 15:42:56.329108 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":438,"threads_started":1}
I20260808 15:42:56.371452 32543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:56.371762 32543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:56.375306 32543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Sending DeleteTablet for 3 replicas of tablet b0422692ac194623916d4504802a1da6
I20260808 15:42:56.376271 32701 tablet_service.cc:1514] Processing DeleteTablet for tablet b0422692ac194623916d4504802a1da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:40742
I20260808 15:42:56.376485 388 tablet_service.cc:1514] Processing DeleteTablet for tablet b0422692ac194623916d4504802a1da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46800
I20260808 15:42:56.377029 542 tablet_service.cc:1514] Processing DeleteTablet for tablet b0422692ac194623916d4504802a1da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:34316
I20260808 15:42:56.404515 726 tablet_replica.cc:330] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: stopping tablet replica
I20260808 15:42:56.405526 726 raft_consensus.cc:2229] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:56.411273 726 raft_consensus.cc:2258] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.411401 728 tablet_replica.cc:330] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: stopping tablet replica
I20260808 15:42:56.411667 728 raft_consensus.cc:2229] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:56.412031 728 raft_consensus.cc:2258] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.412835 727 tablet_replica.cc:330] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: stopping tablet replica
I20260808 15:42:56.413100 727 raft_consensus.cc:2229] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:56.413439 728 ts_tablet_manager.cc:1905] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:56.413419 727 raft_consensus.cc:2258] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.414426 726 ts_tablet_manager.cc:1905] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:56.414583 727 ts_tablet_manager.cc:1905] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:56.421631 727 ts_tablet_manager.cc:1918] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:56.421821 727 log.cc:1196] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/b0422692ac194623916d4504802a1da6
I20260808 15:42:56.422371 727 ts_tablet_manager.cc:1939] T b0422692ac194623916d4504802a1da6 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting consensus metadata
I20260808 15:42:56.423766 32522 catalog_manager.cc:4841] TS 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073): tablet b0422692ac194623916d4504802a1da6 (table test-table [id=89b0f46251244d728baa477262bea691]) successfully deleted
I20260808 15:42:56.430258 726 ts_tablet_manager.cc:1918] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:56.430528 726 log.cc:1196] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/b0422692ac194623916d4504802a1da6
I20260808 15:42:56.431093 726 ts_tablet_manager.cc:1939] T b0422692ac194623916d4504802a1da6 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting consensus metadata
I20260808 15:42:56.434943 728 ts_tablet_manager.cc:1918] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:56.435102 728 log.cc:1196] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/b0422692ac194623916d4504802a1da6
I20260808 15:42:56.435690 728 ts_tablet_manager.cc:1939] T b0422692ac194623916d4504802a1da6 P 040054f4cc4f455aad09b77ee3322e54: Deleting consensus metadata
I20260808 15:42:56.437232 32525 catalog_manager.cc:4841] TS 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629): tablet b0422692ac194623916d4504802a1da6 (table test-table [id=89b0f46251244d728baa477262bea691]) successfully deleted
I20260808 15:42:56.447249 32523 catalog_manager.cc:4841] TS ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303): tablet b0422692ac194623916d4504802a1da6 (table test-table [id=89b0f46251244d728baa477262bea691]) successfully deleted
Aug 08 15:42:56 dist-test-slave-gp6n krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:42:56 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:42:56 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:42:56.511729 736 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:42:56.485868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.23.130.62:42979)
0808 15:42:56.500765 (+ 14897us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.500771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.500790 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:42:56.500848 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.500852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.500913 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.501087 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:56.502890 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.503644 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.505084 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.505763 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.505899 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.511429 (+ 5530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54052: BlockingRecv error: recv got EOF from 127.0.0.1:54052 (error 108)
Metrics: {"server-negotiator.queue_time_us":14784,"thread_start_us":14630,"threads_started":1}
Aug 08 15:42:56 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM
I20260808 15:42:56.528975 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.514883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.23.130.62:42979)
0808 15:42:56.515045 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.515050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.515636 (+ 586us) server_negotiation.cc:407] Connection header received
0808 15:42:56.515820 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.515826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.515899 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.516000 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:56.517175 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.517869 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.519187 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.519600 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.519732 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.523362 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:56.523367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:56.523371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:56.523402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:56.526236 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:56.527002 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:56.527009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:56.527012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:56.527181 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:56.527593 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:56.527597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:56.527599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:56.527841 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:56.528120 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.528390 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.528609 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:42:56.532707 32543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:42:56.533339 32543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:56.540486 388 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 (DEFAULT_TABLE table=test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.540647 32701 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 (DEFAULT_TABLE table=test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.540977 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd049b2e2de4c15b3dd5c7238524818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.541049 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd049b2e2de4c15b3dd5c7238524818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.543071 542 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 (DEFAULT_TABLE table=test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.543534 542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd049b2e2de4c15b3dd5c7238524818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.562978 744 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Bootstrap starting.
I20260808 15:42:56.563856 743 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Bootstrap starting.
I20260808 15:42:56.565240 744 tablet_bootstrap.cc:654] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.565855 743 tablet_bootstrap.cc:654] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.568447 742 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Bootstrap starting.
I20260808 15:42:56.570181 742 tablet_bootstrap.cc:654] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.572399 743 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: No bootstrap required, opened a new log
I20260808 15:42:56.572616 743 ts_tablet_manager.cc:1397] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 15:42:56.572839 742 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: No bootstrap required, opened a new log
I20260808 15:42:56.573010 742 ts_tablet_manager.cc:1397] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 15:42:56.573696 743 raft_consensus.cc:348] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.573792 742 raft_consensus.cc:348] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.574059 742 raft_consensus.cc:374] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.574096 743 raft_consensus.cc:374] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.574182 742 raft_consensus.cc:729] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.574331 743 raft_consensus.cc:729] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.574525 742 consensus_queue.cc:260] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.574677 743 consensus_queue.cc:260] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.575183 742 ts_tablet_manager.cc:1428] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:56.575521 743 ts_tablet_manager.cc:1428] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 15:42:56.577481 744 tablet_bootstrap.cc:492] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: No bootstrap required, opened a new log
I20260808 15:42:56.577663 744 ts_tablet_manager.cc:1397] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260808 15:42:56.578636 744 raft_consensus.cc:348] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.578902 744 raft_consensus.cc:374] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.579010 744 raft_consensus.cc:729] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.579262 744 consensus_queue.cc:260] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.579865 744 ts_tablet_manager.cc:1428] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 15:42:56.765408 657 raft_consensus.cc:482] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:56.765702 657 raft_consensus.cc:504] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.767444 657 leader_election.cc:288] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:56.775648 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.768152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:42281 (local address 127.23.130.3:39629)
0808 15:42:56.768360 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.768367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.769041 (+ 674us) server_negotiation.cc:407] Connection header received
0808 15:42:56.769169 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.769174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.769266 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.769384 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:56.770204 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.770979 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.773418 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.774680 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.774875 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.774962 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.775243 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.775387 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:42:56.775758 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.767769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.23.130.2:42281)
0808 15:42:56.768919 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:56.768934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:56.769039 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:56.769440 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:56.769448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:56.769478 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:56.769947 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.769965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.771125 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.771130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.773237 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.773253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.774889 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.774895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.775078 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.775102 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:42:56.775218 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:42:56.775352 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":3,"thread_start_us":907,"threads_started":1}
I20260808 15:42:56.776531 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.767672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:37303 (local address 127.23.130.2:57439)
0808 15:42:56.768682 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:56.768707 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:56.768953 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:56.769501 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:56.769505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:56.769546 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:42:56.770001 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.770015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.771597 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.771602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.773617 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:56.773627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.775152 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.775159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:56.775347 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.775382 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:42:56.776100 (+ 718us) client_negotiation.cc:240] Negotiation successful
0808 15:42:56.776232 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":54,"thread_start_us":709,"threads_started":1}
I20260808 15:42:56.776532 569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd049b2e2de4c15b3dd5c7238524818" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54" is_pre_election: true
I20260808 15:42:56.776877 569 raft_consensus.cc:2454] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 0.
I20260808 15:42:56.776950 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.767894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:57439 (local address 127.23.130.1:37303)
0808 15:42:56.768198 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.768208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.768812 (+ 604us) server_negotiation.cc:407] Connection header received
0808 15:42:56.769149 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.769153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.769217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.769294 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:42:56.770193 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.771444 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.773779 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.774710 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.774917 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.775001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.775579 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.776079 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:42:56.777468 335 leader_election.cc:302] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, 34e9987be79f4c46b36fb5b42f64aae2; no voters:
I20260808 15:42:56.777588 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd049b2e2de4c15b3dd5c7238524818" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" is_pre_election: true
I20260808 15:42:56.777873 657 raft_consensus.cc:2781] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:56.777886 32727 raft_consensus.cc:2454] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 0.
I20260808 15:42:56.777983 657 raft_consensus.cc:482] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:56.778043 657 raft_consensus.cc:3037] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.780320 657 raft_consensus.cc:504] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.780946 657 leader_election.cc:288] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 election: Requested vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:56.781468 569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd049b2e2de4c15b3dd5c7238524818" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54"
I20260808 15:42:56.781518 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd049b2e2de4c15b3dd5c7238524818" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
I20260808 15:42:56.781700 569 raft_consensus.cc:3037] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.781744 32727 raft_consensus.cc:3037] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:56.783794 569 raft_consensus.cc:2454] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 1.
I20260808 15:42:56.784219 335 leader_election.cc:302] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040054f4cc4f455aad09b77ee3322e54, 34e9987be79f4c46b36fb5b42f64aae2; no voters:
I20260808 15:42:56.784516 657 raft_consensus.cc:2781] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:56.785053 32727 raft_consensus.cc:2454] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 1.
I20260808 15:42:56.785070 657 raft_consensus.cc:686] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 LEADER]: Becoming Leader. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Running, Role: LEADER
I20260808 15:42:56.785511 657 consensus_queue.cc:237] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.789312 32544 catalog_manager.cc:5495] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "34e9987be79f4c46b36fb5b42f64aae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:56.861873 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:56.854507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.23.130.2:41073)
0808 15:42:56.854977 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:56.854982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:56.854999 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:42:56.855088 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:56.855091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:56.855168 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:56.855319 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:56.855874 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.856830 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.859068 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:56.859501 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:56.859602 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:56.860261 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:56.860999 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:56.861229 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:56.861327 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 15:42:56.861475 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":264,"threads_started":1}
I20260808 15:42:56.865813 569 raft_consensus.cc:1261] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Refusing update from remote peer 34e9987be79f4c46b36fb5b42f64aae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:56.866094 32727 raft_consensus.cc:1261] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Refusing update from remote peer 34e9987be79f4c46b36fb5b42f64aae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:56.866624 754 consensus_queue.cc:1035] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.867565 754 consensus_queue.cc:1035] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:56.907912 32544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:56.908188 32544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:56.911868 32544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Sending DeleteTablet for 3 replicas of tablet 4cd049b2e2de4c15b3dd5c7238524818
I20260808 15:42:56.913465 388 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46800
I20260808 15:42:56.913769 727 tablet_replica.cc:330] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: stopping tablet replica
I20260808 15:42:56.913794 542 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:34316
I20260808 15:42:56.913983 727 raft_consensus.cc:2229] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:56.914558 727 raft_consensus.cc:2258] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.914597 728 tablet_replica.cc:330] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: stopping tablet replica
I20260808 15:42:56.914821 728 raft_consensus.cc:2229] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:56.915062 728 raft_consensus.cc:2258] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.915771 727 ts_tablet_manager.cc:1905] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:56.915901 32544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 15:42:56.916147 728 ts_tablet_manager.cc:1905] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 15:42:56.916592 32544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:42:56.918998 32701 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd049b2e2de4c15b3dd5c7238524818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:56 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:40742
I20260808 15:42:56.919323 726 tablet_replica.cc:330] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: stopping tablet replica
I20260808 15:42:56.919708 726 raft_consensus.cc:2229] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:56.919975 726 raft_consensus.cc:2258] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:56.921221 726 ts_tablet_manager.cc:1905] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:56.922928 727 ts_tablet_manager.cc:1918] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:56.923101 727 log.cc:1196] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/4cd049b2e2de4c15b3dd5c7238524818
I20260808 15:42:56.923354 728 ts_tablet_manager.cc:1918] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:56.923496 728 log.cc:1196] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/4cd049b2e2de4c15b3dd5c7238524818
I20260808 15:42:56.923591 727 ts_tablet_manager.cc:1939] T 4cd049b2e2de4c15b3dd5c7238524818 P 34e9987be79f4c46b36fb5b42f64aae2: Deleting consensus metadata
I20260808 15:42:56.923897 728 ts_tablet_manager.cc:1939] T 4cd049b2e2de4c15b3dd5c7238524818 P 040054f4cc4f455aad09b77ee3322e54: Deleting consensus metadata
I20260808 15:42:56.925110 32522 catalog_manager.cc:4841] TS 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073): tablet 4cd049b2e2de4c15b3dd5c7238524818 (table test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]) successfully deleted
I20260808 15:42:56.925565 388 tablet_service.cc:1467] Processing CreateTablet for tablet fec33119a03244c8b170b9437b92233b (DEFAULT_TABLE table=test-table [id=1ce1ddb807d3452da059291cca32da02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.926030 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec33119a03244c8b170b9437b92233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.927428 32525 catalog_manager.cc:4841] TS 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629): tablet 4cd049b2e2de4c15b3dd5c7238524818 (table test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]) successfully deleted
I20260808 15:42:56.927924 726 ts_tablet_manager.cc:1918] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:42:56.928068 726 log.cc:1196] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/4cd049b2e2de4c15b3dd5c7238524818
I20260808 15:42:56.928169 542 tablet_service.cc:1467] Processing CreateTablet for tablet fec33119a03244c8b170b9437b92233b (DEFAULT_TABLE table=test-table [id=1ce1ddb807d3452da059291cca32da02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.928491 726 ts_tablet_manager.cc:1939] T 4cd049b2e2de4c15b3dd5c7238524818 P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting consensus metadata
I20260808 15:42:56.928589 542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec33119a03244c8b170b9437b92233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.931005 32523 catalog_manager.cc:4841] TS ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303): tablet 4cd049b2e2de4c15b3dd5c7238524818 (table test-table [id=314a5eb90e1a4b34bccaa4fd7a5bf2a2]) successfully deleted
I20260808 15:42:56.931566 32701 tablet_service.cc:1467] Processing CreateTablet for tablet fec33119a03244c8b170b9437b92233b (DEFAULT_TABLE table=test-table [id=1ce1ddb807d3452da059291cca32da02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:42:56.931995 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec33119a03244c8b170b9437b92233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:42:56.932780 743 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Bootstrap starting.
I20260808 15:42:56.934643 743 tablet_bootstrap.cc:654] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.941093 744 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Bootstrap starting.
I20260808 15:42:56.942587 742 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Bootstrap starting.
I20260808 15:42:56.942701 744 tablet_bootstrap.cc:654] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.944761 742 tablet_bootstrap.cc:654] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Neither blocks nor log segments found. Creating new log.
I20260808 15:42:56.951839 743 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: No bootstrap required, opened a new log
I20260808 15:42:56.952028 743 ts_tablet_manager.cc:1397] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20260808 15:42:56.952808 742 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: No bootstrap required, opened a new log
I20260808 15:42:56.952777 743 raft_consensus.cc:348] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.953028 743 raft_consensus.cc:374] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.953091 743 raft_consensus.cc:729] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.953317 743 consensus_queue.cc:260] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.953832 743 ts_tablet_manager.cc:1428] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 15:42:56.954118 742 ts_tablet_manager.cc:1397] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260808 15:42:56.955950 742 raft_consensus.cc:348] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.956389 742 raft_consensus.cc:374] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.956557 742 raft_consensus.cc:729] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac418d7dab384f87ab44aaa8f0db0f0e, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.958026 744 tablet_bootstrap.cc:492] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: No bootstrap required, opened a new log
I20260808 15:42:56.958221 744 ts_tablet_manager.cc:1397] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260808 15:42:56.959070 744 raft_consensus.cc:348] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.959371 744 raft_consensus.cc:374] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:42:56.959483 744 raft_consensus.cc:729] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040054f4cc4f455aad09b77ee3322e54, State: Initialized, Role: FOLLOWER
I20260808 15:42:56.959769 744 consensus_queue.cc:260] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.960366 744 ts_tablet_manager.cc:1428] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:42:56.959820 742 consensus_queue.cc:260] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:56.961457 742 ts_tablet_manager.cc:1428] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260808 15:42:57.079066 754 raft_consensus.cc:482] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:42:57.079372 754 raft_consensus.cc:504] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:57.080242 754 leader_election.cc:288] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:57.080794 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec33119a03244c8b170b9437b92233b" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" is_pre_election: true
I20260808 15:42:57.080823 570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec33119a03244c8b170b9437b92233b" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54" is_pre_election: true
I20260808 15:42:57.081054 32727 raft_consensus.cc:2454] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 0.
I20260808 15:42:57.081143 570 raft_consensus.cc:2454] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 0.
I20260808 15:42:57.081588 334 leader_election.cc:302] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e9987be79f4c46b36fb5b42f64aae2, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:57.081965 754 raft_consensus.cc:2781] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:42:57.082113 754 raft_consensus.cc:482] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:42:57.082235 754 raft_consensus.cc:3037] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:57.084604 754 raft_consensus.cc:504] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:57.085208 754 leader_election.cc:288] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 election: Requested vote from peers ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303), 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629)
I20260808 15:42:57.085778 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec33119a03244c8b170b9437b92233b" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e"
I20260808 15:42:57.085796 570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec33119a03244c8b170b9437b92233b" candidate_uuid: "34e9987be79f4c46b36fb5b42f64aae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040054f4cc4f455aad09b77ee3322e54"
I20260808 15:42:57.085999 32727 raft_consensus.cc:3037] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:57.086054 570 raft_consensus.cc:3037] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:42:57.088299 32727 raft_consensus.cc:2454] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 1.
I20260808 15:42:57.088852 334 leader_election.cc:302] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e9987be79f4c46b36fb5b42f64aae2, ac418d7dab384f87ab44aaa8f0db0f0e; no voters:
I20260808 15:42:57.089197 754 raft_consensus.cc:2781] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:42:57.089383 754 raft_consensus.cc:686] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 LEADER]: Becoming Leader. State: Replica: 34e9987be79f4c46b36fb5b42f64aae2, State: Running, Role: LEADER
I20260808 15:42:57.089689 754 consensus_queue.cc:237] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } }
I20260808 15:42:57.089872 570 raft_consensus.cc:2454] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e9987be79f4c46b36fb5b42f64aae2 in term 1.
I20260808 15:42:57.092832 32544 catalog_manager.cc:5495] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "34e9987be79f4c46b36fb5b42f64aae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e9987be79f4c46b36fb5b42f64aae2" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 } health_report { overall_health: UNKNOWN } } }
I20260808 15:42:57.134874 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.128346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:39629 (local address 127.23.130.2:40375)
0808 15:42:57.128963 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:57.128984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:57.129108 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:57.129498 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:57.129503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:57.129514 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:57.129900 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:57.129908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.131383 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.131391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:57.132582 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:57.132596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.133593 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.133601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:57.133837 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.133916 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:57.134336 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:57.134344 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:42:57.134469 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 15:42:57.134579 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
I20260808 15:42:57.134867 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.128482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:40375 (local address 127.23.130.3:39629)
0808 15:42:57.128661 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:57.128668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:57.129070 (+ 402us) server_negotiation.cc:407] Connection header received
0808 15:42:57.129228 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:57.129234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:57.129335 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:57.129431 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:57.130070 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.131247 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.132720 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.133379 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.133521 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.134097 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:57.134240 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:57.134329 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:57.134481 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 15:42:57.134564 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:42:57.142889 570 raft_consensus.cc:1261] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Refusing update from remote peer 34e9987be79f4c46b36fb5b42f64aae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:57.142927 32727 raft_consensus.cc:1261] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Refusing update from remote peer 34e9987be79f4c46b36fb5b42f64aae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:42:57.143492 657 consensus_queue.cc:1035] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040054f4cc4f455aad09b77ee3322e54" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:57.143780 754 consensus_queue.cc:1035] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac418d7dab384f87ab44aaa8f0db0f0e" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:42:57.155265 760 mvcc.cc:205] Tried to move back new op lower bound from 7316290671172333568 to 7316290670962573312. Current Snapshot: MvccSnapshot[applied={T|T < 7316290671172333568}]
I20260808 15:42:57.180018 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.173577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41073 (local address 127.23.130.3:54153)
0808 15:42:57.174240 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:42:57.174257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:42:57.174365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:42:57.174755 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:42:57.174758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:42:57.174767 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:42:57.175006 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:57.175013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.176547 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.176552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:57.177826 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:42:57.177838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.178818 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.178824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:42:57.178972 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.179008 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:42:57.179406 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:42:57.179420 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:42:57.179532 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 15:42:57.179667 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":347,"threads_started":1}
I20260808 15:42:57.179996 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.173581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:54153 (local address 127.23.130.2:41073)
0808 15:42:57.173764 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:57.173772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:57.174349 (+ 577us) server_negotiation.cc:407] Connection header received
0808 15:42:57.174534 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:57.174540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:57.174629 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:57.174732 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:42:57.175146 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.176394 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.177980 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.178651 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.178811 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.179175 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:42:57.179298 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:42:57.179405 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:57.179560 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 15:42:57.179639 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:57.217012 32544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:57.217252 32544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:42:57.220109 32544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94344fa4a9b4461b9035c467a560d6c: Sending DeleteTablet for 3 replicas of tablet fec33119a03244c8b170b9437b92233b
I20260808 15:42:57.221124 32701 tablet_service.cc:1514] Processing DeleteTablet for tablet fec33119a03244c8b170b9437b92233b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:57 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:40742
I20260808 15:42:57.221230 388 tablet_service.cc:1514] Processing DeleteTablet for tablet fec33119a03244c8b170b9437b92233b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:57 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46800
I20260808 15:42:57.221539 727 tablet_replica.cc:330] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: stopping tablet replica
I20260808 15:42:57.221576 726 tablet_replica.cc:330] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: stopping tablet replica
I20260808 15:42:57.221751 726 raft_consensus.cc:2229] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:57.221761 727 raft_consensus.cc:2229] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:42:57.221870 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 32621
I20260808 15:42:57.221963 726 raft_consensus.cc:2258] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:57.222126 727 raft_consensus.cc:2258] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:57.223202 727 ts_tablet_manager.cc:1905] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:57.224300 726 ts_tablet_manager.cc:1905] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:57.226294 542 tablet_service.cc:1514] Processing DeleteTablet for tablet fec33119a03244c8b170b9437b92233b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:42:57 UTC) from {username='oryx', principal='oryx/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:34316
I20260808 15:42:57.226634 728 tablet_replica.cc:330] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: stopping tablet replica
I20260808 15:42:57.226979 728 raft_consensus.cc:2229] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:42:57.227296 728 raft_consensus.cc:2258] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:42:57.228605 728 ts_tablet_manager.cc:1905] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:42:57.232604 727 ts_tablet_manager.cc:1918] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:57.232779 727 log.cc:1196] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/fec33119a03244c8b170b9437b92233b
I20260808 15:42:57.233366 727 ts_tablet_manager.cc:1939] T fec33119a03244c8b170b9437b92233b P 34e9987be79f4c46b36fb5b42f64aae2: Deleting consensus metadata
I20260808 15:42:57.234848 32522 catalog_manager.cc:4841] TS 34e9987be79f4c46b36fb5b42f64aae2 (127.23.130.2:41073): tablet fec33119a03244c8b170b9437b92233b (table test-table [id=1ce1ddb807d3452da059291cca32da02]) successfully deleted
I20260808 15:42:57.241007 726 ts_tablet_manager.cc:1918] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:57.242129 728 ts_tablet_manager.cc:1918] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:42:57.242352 728 log.cc:1196] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/fec33119a03244c8b170b9437b92233b
I20260808 15:42:57.242601 726 log.cc:1196] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/fec33119a03244c8b170b9437b92233b
I20260808 15:42:57.243091 728 ts_tablet_manager.cc:1939] T fec33119a03244c8b170b9437b92233b P 040054f4cc4f455aad09b77ee3322e54: Deleting consensus metadata
I20260808 15:42:57.243119 726 ts_tablet_manager.cc:1939] T fec33119a03244c8b170b9437b92233b P ac418d7dab384f87ab44aaa8f0db0f0e: Deleting consensus metadata
Aug 08 15:42:57 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.1@KRBTEST.COM
I20260808 15:42:57.244341 32523 catalog_manager.cc:4841] TS ac418d7dab384f87ab44aaa8f0db0f0e (127.23.130.1:37303): tablet fec33119a03244c8b170b9437b92233b (table test-table [id=1ce1ddb807d3452da059291cca32da02]) successfully deleted
I20260808 15:42:57.244422 32525 catalog_manager.cc:4841] TS 040054f4cc4f455aad09b77ee3322e54 (127.23.130.3:39629): tablet fec33119a03244c8b170b9437b92233b (table test-table [id=1ce1ddb807d3452da059291cca32da02]) successfully deleted
I20260808 15:42:57.253692 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.222900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.23.130.1:37303)
0808 15:42:57.223095 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:57.223102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:57.235037 (+ 11935us) server_negotiation.cc:407] Connection header received
0808 15:42:57.235248 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:57.235254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:57.235336 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:57.235460 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:57.236694 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.237439 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.239113 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.239642 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.239819 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.244976 (+ 5157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:57.244994 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:57.245014 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:57.245056 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:57.249260 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.250232 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.250245 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.250251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.250432 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.250992 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.250997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.251002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.251459 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:57.251792 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:57.253094 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0808 15:42:57.253338 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:57.254618 302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:57.387595 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 32621
I20260808 15:42:57.413477 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 313
Aug 08 15:42:57 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.2@KRBTEST.COM
I20260808 15:42:57.429342 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.414499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.23.130.2:41073)
0808 15:42:57.414714 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:57.414720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:57.414742 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:42:57.414823 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:57.414828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:57.414887 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:57.415008 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:57.416266 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.417006 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.418286 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.418858 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.419005 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.423249 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:57.423262 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:57.423277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:57.423310 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:57.426301 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.427192 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.427202 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.427207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.427342 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.427727 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.427733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.427736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.428114 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:57.428431 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:57.428720 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:42:57.428952 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:42:57.430088 460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:57.556895 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 313
I20260808 15:42:57.582916 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 470
Aug 08 15:42:57 dist-test-slave-gp6n krb5kdc[32490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.130.3@KRBTEST.COM
I20260808 15:42:57.598009 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:42:57.583831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.23.130.3:39629)
0808 15:42:57.584003 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:42:57.584007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:42:57.584025 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:42:57.584093 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:42:57.584097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:42:57.584169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:42:57.584255 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:42:57.585532 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.586288 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.587653 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:42:57.588099 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:42:57.588199 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:42:57.592546 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:42:57.592564 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:42:57.592581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:42:57.592617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:42:57.595489 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.596106 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.596114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.596116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.596242 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:42:57.596591 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:42:57.596595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:42:57.596598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:42:57.596974 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:42:57.597226 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 15:42:57.597450 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 15:42:57.597636 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:42:57.598693 615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:57.746845 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 470
I20260808 15:42:57.779058 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 32503
I20260808 15:42:57.780335 32587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:42:57.899212 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 32503
2026-08-08T15:42:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7661 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:58 dist-test-slave-gp6n krb5kdc[791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:58 dist-test-slave-gp6n krb5kdc[791](info): set up 2 sockets
Aug 08 15:42:58 dist-test-slave-gp6n krb5kdc[791](info): commencing operation
krb5kdc: starting...
W20260808 15:42:59.149927 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:42:59 dist-test-slave-gp6n krb5kdc[791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:42:59.424388 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:35517
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:42:59.762827 807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:42:59.763105 807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:42:59.763288 807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:42:59.770511 807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:42:59.770615 807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:42:59.770714 807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:42:59.770816 807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:42:59.770910 807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:42:59.780689 807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:42:59.782166 807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:42:59.783542 807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:42:59.795624 816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:59.795712 814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:42:59.795598 813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:42:59.797073 807 server_base.cc:1034] running on GCE node
I20260808 15:42:59.797868 807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:42:59.799351 807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:42:59.800557 807 hybrid_clock.cc:648] HybridClock initialized: now 1786203779800513 us; error 32 us; skew 500 ppm
I20260808 15:42:59.800909 807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T15:42:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1970 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:42:59 dist-test-slave-gp6n krb5kdc[824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:42:59 dist-test-slave-gp6n krb5kdc[824](info): set up 2 sockets
Aug 08 15:42:59 dist-test-slave-gp6n krb5kdc[824](info): commencing operation
krb5kdc: starting...
W20260808 15:43:01.137735 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:43:01 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:43:01.409941 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39349
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:01.744148 837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:01.744431 837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:01.744522 837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:01.751703 837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:43:01.751823 837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:01.751873 837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:01.751924 837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:43:01.751969 837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:43:01.761821 837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:01.763096 837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:01.764469 837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:01.776312 842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:01.776266 843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:01.776748 845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:01.777599 837 server_base.cc:1034] running on GCE node
I20260808 15:43:01.778280 837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:01.779635 837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:01.780836 837 hybrid_clock.cc:648] HybridClock initialized: now 1786203781780787 us; error 37 us; skew 500 ppm
I20260808 15:43:01.781181 837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:01 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:01.790177 837 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:43:01.791844 837 webserver.cc:460] Webserver started at http://127.23.130.62:38637/ using document root <none> and password file <none>
I20260808 15:43:01.792547 837 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:01.792687 837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:01.793012 837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:01.796057 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "439199b3d3c24971a205b329b94241c0"
format_stamp: "Formatted at 2026-08-08 15:43:01 on dist-test-slave-gp6n"
server_key: "d1b5f57db215b062d9035d6fdeb3271a"
server_key_iv: "8689e2e4127db6a621350261285ea4e5"
server_key_version: "encryptionkey@0"
I20260808 15:43:01.796850 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "439199b3d3c24971a205b329b94241c0"
format_stamp: "Formatted at 2026-08-08 15:43:01 on dist-test-slave-gp6n"
server_key: "d1b5f57db215b062d9035d6fdeb3271a"
server_key_iv: "8689e2e4127db6a621350261285ea4e5"
server_key_version: "encryptionkey@0"
I20260808 15:43:01.802052 837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:43:01.805503 853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:01.806794 837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:01.806986 837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "439199b3d3c24971a205b329b94241c0"
format_stamp: "Formatted at 2026-08-08 15:43:01 on dist-test-slave-gp6n"
server_key: "d1b5f57db215b062d9035d6fdeb3271a"
server_key_iv: "8689e2e4127db6a621350261285ea4e5"
server_key_version: "encryptionkey@0"
I20260808 15:43:01.807178 837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:01.820906 837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:01.824854 837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:01.825158 837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:01.843726 837 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:39349
I20260808 15:43:01.843746 907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:39349 every 8 connection(s)
I20260808 15:43:01.845216 837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:43:01.849117 909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:01.855705 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 837
I20260808 15:43:01.855988 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:43:01.856820 24072 external_mini_cluster.cc:1428] Setting key fb9fdf57983f9a48f3297745f4990d30
I20260808 15:43:01.860585 909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Bootstrap starting.
I20260808 15:43:01.864188 909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:01.865294 909 log.cc:824] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:01.868289 909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: No bootstrap required, opened a new log
Aug 08 15:43:01 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:01.876716 909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:01.877081 909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:01.877202 909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439199b3d3c24971a205b329b94241c0, State: Initialized, Role: FOLLOWER
I20260808 15:43:01.877754 909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:01.878023 909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:43:01.878175 909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:43:01.878386 909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:01.880193 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:01.859009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.23.130.62:39349)
0808 15:43:01.859779 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:01.859791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:01.859837 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:43:01.861637 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:01.861659 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:01.862058 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:01.862382 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:01.864324 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:01.865786 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:01.867400 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:01.868033 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:01.868159 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:01.872857 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:01.872869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:01.872883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:01.872923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:01.876092 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:01.876745 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:01.876752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:01.876757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:01.876926 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:01.877357 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:01.877377 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:01.877379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:01.878015 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:01.878340 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:01.878791 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:43:01.879169 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":375,"threads_started":1}
I20260808 15:43:01.880945 909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:01.881443 909 leader_election.cc:302] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439199b3d3c24971a205b329b94241c0; no voters:
I20260808 15:43:01.882123 909 leader_election.cc:288] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:43:01.882233 915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:43:01.883870 915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 LEADER]: Becoming Leader. State: Replica: 439199b3d3c24971a205b329b94241c0, State: Running, Role: LEADER
I20260808 15:43:01.884243 909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:43:01.884372 915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:01.888051 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439199b3d3c24971a205b329b94241c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } } }
I20260808 15:43:01.888038 917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439199b3d3c24971a205b329b94241c0. Latest consensus state: current_term: 1 leader_uuid: "439199b3d3c24971a205b329b94241c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } } }
I20260808 15:43:01.888490 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:01.888516 917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:01.890966 925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:43:01.896045 925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:43:01.905483 925 catalog_manager.cc:1347] Generated new cluster ID: 94ffdf7c03c7492e8e35213a99e372f6
I20260808 15:43:01.905611 925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:43:01.926244 925 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:43:01.927608 925 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:43:01.934568 925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Generated new TSK 0
I20260808 15:43:01.935391 925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:43:02.010016 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:02.372267 941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:02.372532 941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:02.372627 941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:02.379743 941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:02.379846 941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:02.380046 941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:43:02.389798 941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:02.391105 941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:02.392478 941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:02.405563 947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:02.405617 948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:02.405650 951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:02.406364 941 server_base.cc:1034] running on GCE node
I20260808 15:43:02.407006 941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:02.407920 941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:02.409144 941 hybrid_clock.cc:648] HybridClock initialized: now 1786203782409100 us; error 44 us; skew 500 ppm
I20260808 15:43:02.409469 941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:02 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203782, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:02.412848 941 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:43:02.414251 941 webserver.cc:460] Webserver started at http://127.23.130.1:46381/ using document root <none> and password file <none>
I20260808 15:43:02.414964 941 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:02.415094 941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:02.415411 941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:02.418295 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "c05c65c2afb8437485c302a74b9fb65a"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "482d9dc497b88e412017a0ba1a128358"
server_key_iv: "0e53b174c69d937fe747536c13c6eaf0"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.419190 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "c05c65c2afb8437485c302a74b9fb65a"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "482d9dc497b88e412017a0ba1a128358"
server_key_iv: "0e53b174c69d937fe747536c13c6eaf0"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.424256 941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 15:43:02.427666 958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:02.428911 941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:43:02.429071 941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "c05c65c2afb8437485c302a74b9fb65a"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "482d9dc497b88e412017a0ba1a128358"
server_key_iv: "0e53b174c69d937fe747536c13c6eaf0"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.429247 941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:02.441658 941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:02.445598 941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:02.445875 941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:02.446841 941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:02.448128 941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:02.448204 941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:02.448338 941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:02.448397 941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:02.482115 941 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:46209
I20260808 15:43:02.482204 1082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:46209 every 8 connection(s)
I20260808 15:43:02.483441 941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:43:02.490775 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 941
I20260808 15:43:02.491006 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:43:02.491624 24072 external_mini_cluster.cc:1428] Setting key 6207b7eebd92a46b0a3d8a903038a972
Aug 08 15:43:02 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203782, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:02.504429 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:02.486310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:50551 (local address 127.23.130.62:39349)
0808 15:43:02.486923 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:02.486928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:02.487807 (+ 879us) server_negotiation.cc:407] Connection header received
0808 15:43:02.489664 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:02.489670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:02.489781 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:02.489904 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:02.491593 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:02.492543 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:02.494290 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:02.494782 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:02.494922 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:02.498726 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:02.498731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:02.498734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:02.498765 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:02.501326 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:02.502060 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:02.502065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:02.502069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:02.502193 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:02.502686 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:02.502691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:02.502694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:02.502964 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:02.503119 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:02.503828 (+ 709us) server_negotiation.cc:299] Negotiation successful
0808 15:43:02.504065 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":290,"threads_started":1}
I20260808 15:43:02.505604 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:02.486841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:50551)
0808 15:43:02.487618 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:02.487666 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:02.489386 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:02.490145 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:02.490154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:02.490760 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:02.491422 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:02.491441 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:02.492707 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:02.492711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:02.494149 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:02.494161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:02.495011 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:02.495021 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:02.495264 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:02.495980 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:02.496003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:02.498509 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:02.501475 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:02.501485 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:02.501502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:02.501935 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:02.502302 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:02.502308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:02.502311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:02.502568 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:02.503092 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:02.503100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:02.503457 (+ 357us) client_negotiation.cc:769] Sending connection context
0808 15:43:02.503716 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 15:43:02.504249 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":304,"threads_started":1}
I20260808 15:43:02.507225 1083 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39349
I20260808 15:43:02.507637 1083 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:02.508661 1083 heartbeater.cc:511] Master 127.23.130.62:39349 requested a full tablet report, sending...
I20260808 15:43:02.511075 870 ts_manager.cc:194] Registered new tserver with Master: c05c65c2afb8437485c302a74b9fb65a (127.23.130.1:46209)
I20260808 15:43:02.512564 870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:50551
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:43:02.553354 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:02.882946 1089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:02.883211 1089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:02.883308 1089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:02.890806 1089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:02.890911 1089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:02.891022 1089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:43:02.900549 1089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:02.901753 1089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:02.903100 1089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:02.917022 1098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:02.917058 1096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:02.917098 1095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:02.917979 1089 server_base.cc:1034] running on GCE node
I20260808 15:43:02.918642 1089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:02.919559 1089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:02.920796 1089 hybrid_clock.cc:648] HybridClock initialized: now 1786203782920752 us; error 51 us; skew 500 ppm
I20260808 15:43:02.921110 1089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:02 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203782, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:02.924407 1089 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:43:02.925877 1089 webserver.cc:460] Webserver started at http://127.23.130.2:40081/ using document root <none> and password file <none>
I20260808 15:43:02.926697 1089 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:02.926836 1089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:02.927160 1089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:02.930035 1089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "2a4cbfe0476142e0a063d154b6925c20"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "4b794fa93670f75b3c645bc22e822bc0"
server_key_iv: "64b86be3d26749badddc321c953c24f2"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.930835 1089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4cbfe0476142e0a063d154b6925c20"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "4b794fa93670f75b3c645bc22e822bc0"
server_key_iv: "64b86be3d26749badddc321c953c24f2"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.936118 1089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:43:02.939579 1105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:02.940912 1089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:43:02.941076 1089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "2a4cbfe0476142e0a063d154b6925c20"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "4b794fa93670f75b3c645bc22e822bc0"
server_key_iv: "64b86be3d26749badddc321c953c24f2"
server_key_version: "encryptionkey@0"
I20260808 15:43:02.941280 1089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:02.960273 1089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:02.964329 1089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:02.964645 1089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:02.965579 1089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:02.967060 1089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:02.967159 1089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:02.967303 1089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:02.967381 1089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:03.001358 1089 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41535
I20260808 15:43:03.001399 1226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41535 every 8 connection(s)
I20260808 15:43:03.002750 1089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:43:03.007851 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1089
I20260808 15:43:03.008068 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:43:03.008610 24072 external_mini_cluster.cc:1428] Setting key 615365831c5add71164e71e804a801ea
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203782, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:03.025813 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.005220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:34021 (local address 127.23.130.62:39349)
0808 15:43:03.005652 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:03.005657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:03.006584 (+ 927us) server_negotiation.cc:407] Connection header received
0808 15:43:03.008905 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:03.008909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:03.008988 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:03.009083 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:03.011026 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.011779 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.014317 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.014799 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.014933 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.019472 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:03.019481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:03.019489 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:03.019537 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:03.022816 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.023532 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.023537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.023545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.023667 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.024092 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.024097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.024104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.024363 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:03.024526 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:03.025200 (+ 674us) server_negotiation.cc:299] Negotiation successful
0808 15:43:03.025423 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260808 15:43:03.026814 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.005702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:34021)
0808 15:43:03.006388 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:03.006483 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:03.008603 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:03.009353 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:03.009361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:03.010054 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:03.010828 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:03.010854 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.011956 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.011965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:03.014160 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:03.014177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.015050 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.015059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:03.015400 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.016140 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:03.016181 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:03.019226 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:03.022966 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:03.022975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:03.022989 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:03.023398 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:03.023779 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:03.023784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:03.023787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:03.023974 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:03.024500 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:03.024507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:03.024832 (+ 325us) client_negotiation.cc:769] Sending connection context
0808 15:43:03.025099 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 15:43:03.025616 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260808 15:43:03.028335 1227 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39349
I20260808 15:43:03.028657 1227 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:03.029407 1227 heartbeater.cc:511] Master 127.23.130.62:39349 requested a full tablet report, sending...
I20260808 15:43:03.031056 870 ts_manager.cc:194] Registered new tserver with Master: 2a4cbfe0476142e0a063d154b6925c20 (127.23.130.2:41535)
I20260808 15:43:03.032369 870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:34021
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:43:03.069490 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:03.411757 1234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:03.412093 1234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:03.412264 1234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:03.419476 1234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:03.419632 1234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:03.419804 1234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:43:03.429325 1234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:03.430660 1234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:03.431965 1234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:03.445336 1239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:03.445372 1240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:03.445425 1242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:03.446182 1234 server_base.cc:1034] running on GCE node
I20260808 15:43:03.446846 1234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:03.447732 1234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:03.448954 1234 hybrid_clock.cc:648] HybridClock initialized: now 1786203783448895 us; error 53 us; skew 500 ppm
I20260808 15:43:03.449255 1234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203783, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:03.452337 1234 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:43:03.453688 1234 webserver.cc:460] Webserver started at http://127.23.130.3:35221/ using document root <none> and password file <none>
I20260808 15:43:03.454358 1234 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:03.454521 1234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:03.454829 1234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:03.457648 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "a77af77c07054986b148f5e5522992f7"
format_stamp: "Formatted at 2026-08-08 15:43:03 on dist-test-slave-gp6n"
server_key: "55a388ed937c7ff5b389057128960d7c"
server_key_iv: "f3a9d691629ca152b31b9643d0da0639"
server_key_version: "encryptionkey@0"
I20260808 15:43:03.458503 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "a77af77c07054986b148f5e5522992f7"
format_stamp: "Formatted at 2026-08-08 15:43:03 on dist-test-slave-gp6n"
server_key: "55a388ed937c7ff5b389057128960d7c"
server_key_iv: "f3a9d691629ca152b31b9643d0da0639"
server_key_version: "encryptionkey@0"
I20260808 15:43:03.463506 1234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:43:03.466792 1251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:03.468055 1234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:43:03.468221 1234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "a77af77c07054986b148f5e5522992f7"
format_stamp: "Formatted at 2026-08-08 15:43:03 on dist-test-slave-gp6n"
server_key: "55a388ed937c7ff5b389057128960d7c"
server_key_iv: "f3a9d691629ca152b31b9643d0da0639"
server_key_version: "encryptionkey@0"
I20260808 15:43:03.468413 1234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:03.486440 1234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:03.490684 1234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:03.490932 1234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:03.491784 1234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:03.493081 1234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:03.493155 1234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:03.493250 1234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:03.493297 1234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:03.516487 1083 heartbeater.cc:503] Master 127.23.130.62:39349 was elected leader, sending a full tablet report...
I20260808 15:43:03.527616 1234 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:36463
I20260808 15:43:03.527633 1374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:36463 every 8 connection(s)
I20260808 15:43:03.528878 1234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:43:03.535001 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1234
I20260808 15:43:03.535220 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:43:03.535775 24072 external_mini_cluster.cc:1428] Setting key 7f89a2c7b95655df99a32f5b02bc2756
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203783, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:03.549386 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.531354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:50093 (local address 127.23.130.62:39349)
0808 15:43:03.531838 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:03.531843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:03.532735 (+ 892us) server_negotiation.cc:407] Connection header received
0808 15:43:03.534723 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:03.534728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:03.534814 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:03.534917 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:03.536948 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.538021 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.539802 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.540194 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.540291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.543848 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:03.543853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:03.543856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:03.543888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:03.546713 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.547400 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.547403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.547405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.547554 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.547927 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.547930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.547932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.548183 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:03.548297 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:03.548934 (+ 637us) server_negotiation.cc:299] Negotiation successful
0808 15:43:03.549093 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":274,"threads_started":1}
I20260808 15:43:03.550766 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.531825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:50093)
0808 15:43:03.532543 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:03.532601 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:03.534439 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:03.535215 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:03.535228 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:03.535966 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:03.536766 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:03.536791 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.538171 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.538175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:03.539678 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:03.539689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.540416 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.540424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:03.540672 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.541313 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:03.541335 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:03.543663 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:03.546850 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:03.546856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:03.546868 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:03.547295 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:03.547642 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:03.547645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:03.547648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:03.547832 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:03.548307 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:03.548313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:03.548628 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 15:43:03.548851 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 15:43:03.549364 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":283,"threads_started":1}
I20260808 15:43:03.552313 1375 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39349
I20260808 15:43:03.552618 1375 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:03.553331 1375 heartbeater.cc:511] Master 127.23.130.62:39349 requested a full tablet report, sending...
I20260808 15:43:03.554805 870 ts_manager.cc:194] Registered new tserver with Master: a77af77c07054986b148f5e5522992f7 (127.23.130.3:36463)
I20260808 15:43:03.555626 870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:50093
I20260808 15:43:03.563925 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
Entry for principal oryx/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab.
WARNING: no policy specified for oryx/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
Entry for principal oryx/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab.
WARNING: no policy specified for oryx/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
Entry for principal oryx/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab.
I20260808 15:43:03.729096 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 941
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:43:03.747670 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.730286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.23.130.1:46209)
0808 15:43:03.732790 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:03.732802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:03.732840 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:43:03.732995 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:03.733003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:03.733286 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:03.733449 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:03.735083 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.736157 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.737406 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.737981 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.738082 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.742209 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:03.742220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:03.742234 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:03.742270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:03.745057 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.745811 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.745817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.745820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.745946 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.746268 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.746271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.746273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.746666 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:03.746885 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:03.747165 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 15:43:03.747360 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":2116,"threads_started":1}
I20260808 15:43:03.748978 1077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:03.832188 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 941
I20260808 15:43:03.854523 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1089
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:43:03.871925 1390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.855522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.23.130.2:41535)
0808 15:43:03.856340 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:03.856349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:03.856376 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:43:03.856447 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:03.856453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:03.856670 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:03.856865 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:03.858272 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.859547 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.860960 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.861581 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.861729 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.866482 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:03.866504 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:03.866524 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:03.866581 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:03.869294 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.869945 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.869952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.869954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.870081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.870441 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.870454 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.870461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.870793 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:03.871031 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:03.871349 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 15:43:03.871625 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":510,"threads_started":1}
I20260808 15:43:03.873216 1222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:03.955545 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1089
I20260808 15:43:03.974102 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1234
Aug 08 15:43:03 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:43:03.990294 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:03.975163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.23.130.3:36463)
0808 15:43:03.975673 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:03.975684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:03.975723 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:43:03.975809 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:03.975817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:03.976135 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:03.976334 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:03.977693 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.978849 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.980068 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:03.980604 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:03.980700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:03.984960 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:03.984971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:03.984987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:03.985019 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:03.987961 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.988566 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.988573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.988575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.988691 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:03.989007 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:03.989010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:03.989012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:03.989356 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:03.989577 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:03.989793 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 15:43:03.989987 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":206,"threads_started":1}
I20260808 15:43:03.991619 1370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:04.074801 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1234
I20260808 15:43:04.094811 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 837
I20260808 15:43:04.096087 903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:04.201699 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 837
I20260808 15:43:04.227752 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39349
--webserver_interface=127.23.130.62
--webserver_port=38637
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:04.571321 1397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:04.571614 1397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:04.571779 1397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:04.578800 1397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:43:04.578900 1397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:04.579005 1397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:04.579101 1397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:43:04.579180 1397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:43:04.588693 1397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=38637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:04.590023 1397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:04.591354 1397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:04.603163 1403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:04.603125 1404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:04.603214 1406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:04.604259 1397 server_base.cc:1034] running on GCE node
I20260808 15:43:04.605029 1397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:04.606465 1397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:04.607733 1397 hybrid_clock.cc:648] HybridClock initialized: now 1786203784607644 us; error 41 us; skew 500 ppm
I20260808 15:43:04.608249 1397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:04 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203784, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:04.611370 1397 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:43:04.613005 1397 webserver.cc:460] Webserver started at http://127.23.130.62:38637/ using document root <none> and password file <none>
I20260808 15:43:04.613840 1397 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:04.614006 1397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:04.619639 1397 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:43:04.622510 1413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:04.623795 1397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:04.623971 1397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "439199b3d3c24971a205b329b94241c0"
format_stamp: "Formatted at 2026-08-08 15:43:01 on dist-test-slave-gp6n"
server_key: "d1b5f57db215b062d9035d6fdeb3271a"
server_key_iv: "8689e2e4127db6a621350261285ea4e5"
server_key_version: "encryptionkey@0"
I20260808 15:43:04.624964 1397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:04.639926 1397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:04.644187 1397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:04.644465 1397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:04.662685 1397 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:39349
I20260808 15:43:04.662706 1473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:39349 every 8 connection(s)
I20260808 15:43:04.664173 1397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:43:04.671012 1474 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:43:04.672758 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1397
I20260808 15:43:04.673743 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:46209
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:39349
--webserver_port=46381
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:43:04.675239 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Bootstrap starting.
I20260808 15:43:04.697211 1474 log.cc:824] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:04.706137 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:43:04.706736 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Bootstrap complete.
I20260808 15:43:04.716025 1474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:04.716909 1474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439199b3d3c24971a205b329b94241c0, State: Initialized, Role: FOLLOWER
I20260808 15:43:04.717556 1474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:04.717805 1474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:43:04.717932 1474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:43:04.718115 1474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:43:04.721956 1474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:04.722500 1474 leader_election.cc:302] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439199b3d3c24971a205b329b94241c0; no voters:
I20260808 15:43:04.723156 1474 leader_election.cc:288] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:43:04.723331 1478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:43:04.731734 1478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [term 2 LEADER]: Becoming Leader. State: Replica: 439199b3d3c24971a205b329b94241c0, State: Running, Role: LEADER
I20260808 15:43:04.732216 1474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:43:04.732383 1478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } }
I20260808 15:43:04.735352 1479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "439199b3d3c24971a205b329b94241c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } } }
I20260808 15:43:04.735816 1479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:04.740643 1480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439199b3d3c24971a205b329b94241c0. Latest consensus state: current_term: 2 leader_uuid: "439199b3d3c24971a205b329b94241c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439199b3d3c24971a205b329b94241c0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39349 } } }
I20260808 15:43:04.741760 1480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:04.746084 1484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:43:04.752295 1484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:43:04.754077 1484 catalog_manager.cc:1259] Loaded cluster ID: 94ffdf7c03c7492e8e35213a99e372f6
I20260808 15:43:04.754290 1484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:43:04.759717 1484 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:43:04.761943 1484 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 439199b3d3c24971a205b329b94241c0: Loaded TSK: 0
I20260808 15:43:04.764089 1484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:43:04.766752 1496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260808 15:43:05.050031 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:05.050309 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:05.050468 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:05.057965 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:05.058082 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:05.058262 1476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:43:05.068336 1476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.1.keytab
--principal=oryx/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:46209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=46381
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:05.069746 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:05.071163 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:05.086082 1502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:05.086143 1505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:05.086771 1501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:05.087399 1476 server_base.cc:1034] running on GCE node
I20260808 15:43:05.088023 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:05.088979 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:05.090210 1476 hybrid_clock.cc:648] HybridClock initialized: now 1786203785090149 us; error 57 us; skew 500 ppm
I20260808 15:43:05.090583 1476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203785, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:05.094077 1476 init.cc:377] Logged in from keytab as oryx/127.23.130.1@KRBTEST.COM (short username oryx)
I20260808 15:43:05.095647 1476 webserver.cc:460] Webserver started at http://127.23.130.1:46381/ using document root <none> and password file <none>
I20260808 15:43:05.096346 1476 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:05.096508 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:05.101956 1476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:43:05.104959 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.106393 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:43:05.106621 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "c05c65c2afb8437485c302a74b9fb65a"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "482d9dc497b88e412017a0ba1a128358"
server_key_iv: "0e53b174c69d937fe747536c13c6eaf0"
server_key_version: "encryptionkey@0"
I20260808 15:43:05.107590 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:05.137550 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:05.141634 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:05.141927 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:05.142947 1476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:05.144304 1476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:05.144444 1476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.144595 1476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:05.144680 1476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.189347 1476 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:46209
I20260808 15:43:05.189422 1640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:46209 every 8 connection(s)
I20260808 15:43:05.191336 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:43:05.194716 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1476
I20260808 15:43:05.195523 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:41535
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:39349
--webserver_port=40081
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:43:05.552156 1645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:05.552426 1645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:05.552582 1645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:05.559648 1645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:05.559765 1645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:05.559952 1645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:43:05.569255 1645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.2.keytab
--principal=oryx/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:41535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=40081
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:05.570593 1645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:05.571892 1645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:05.585752 1651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:05.585752 1652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:05.586534 1654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:05.587464 1645 server_base.cc:1034] running on GCE node
I20260808 15:43:05.588537 1645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:05.590188 1645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:05.591542 1645 hybrid_clock.cc:648] HybridClock initialized: now 1786203785591407 us; error 101 us; skew 500 ppm
I20260808 15:43:05.592085 1645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203785, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:05.597592 1645 init.cc:377] Logged in from keytab as oryx/127.23.130.2@KRBTEST.COM (short username oryx)
I20260808 15:43:05.599943 1645 webserver.cc:460] Webserver started at http://127.23.130.2:40081/ using document root <none> and password file <none>
I20260808 15:43:05.601189 1645 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:05.601397 1645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:05.611327 1645 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 15:43:05.615746 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.617918 1645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:43:05.618144 1645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "2a4cbfe0476142e0a063d154b6925c20"
format_stamp: "Formatted at 2026-08-08 15:43:02 on dist-test-slave-gp6n"
server_key: "4b794fa93670f75b3c645bc22e822bc0"
server_key_iv: "64b86be3d26749badddc321c953c24f2"
server_key_version: "encryptionkey@0"
I20260808 15:43:05.620157 1645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:05.698063 1645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:05.702469 1645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:05.702751 1645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:05.703812 1645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:05.705292 1645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:05.705387 1645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.705530 1645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:05.705619 1645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:05.754487 1645 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41535
I20260808 15:43:05.754544 1787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41535 every 8 connection(s)
I20260808 15:43:05.756042 1645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:43:05.760617 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1645
I20260808 15:43:05.761603 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:36463
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:39349
--webserver_port=35221
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:05 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:43:06.417822 1644 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.207s user 0.011s sys 0.008s
I20260808 15:43:06.419986 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:05.194948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:37699)
0808 15:43:05.196277 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:05.196330 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:05.199650 (+ 3320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:05.201913 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:05.201925 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:05.202764 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:05.203621 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:05.203652 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.206014 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.206022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:05.208059 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:05.208076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.209905 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.209914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:05.210259 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:05.211114 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:05.211148 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:06.418754 (+1207606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":752,"threads_started":1}
W20260808 15:43:06.419981 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:05.194856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:37699 (local address 127.23.130.62:39349)
0808 15:43:05.195985 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:05.196002 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:05.198288 (+ 2286us) server_negotiation.cc:407] Connection header received
0808 15:43:05.200357 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:05.200377 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:05.200825 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:05.201227 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:05.203860 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.205775 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.208241 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.209341 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.209499 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.419188 (+1209689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:37699: BlockingRecv error: recv got EOF from 127.23.130.1:37699 (error 108)
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":297,"threads_started":1}
W20260808 15:43:06.421169 1641 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:39349 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.130.62:39349: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:06.431025 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:06.421890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:47329)
0808 15:43:06.422156 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:06.422181 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:06.422326 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:06.422689 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:06.422694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:06.423081 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:06.423255 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.423263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.424300 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.424305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.425274 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.425286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.425997 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.426003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.426177 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.426829 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:06.426847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:06.430717 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260808 15:43:06.431277 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:06.421942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:47329 (local address 127.23.130.62:39349)
0808 15:43:06.422117 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:06.422124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:06.422255 (+ 131us) server_negotiation.cc:407] Connection header received
0808 15:43:06.422478 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:06.422484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:06.422555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:06.422659 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:06.423402 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.424149 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.425434 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.425821 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.425967 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.431038 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:47329: BlockingRecv error: recv got EOF from 127.23.130.1:47329 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 15:43:06.431494 1793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:06.432112 1793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:06.432356 1793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:06.440603 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:06.431881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:44519)
0808 15:43:06.432058 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:06.432077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:06.432661 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:06.432979 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:06.432984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:06.433314 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:06.433492 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.433505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.434662 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.434667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.435623 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.435635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.436337 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.436343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.436547 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.437128 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:06.437144 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:06.440316 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:43:06.440752 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:06.432324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:44519 (local address 127.23.130.62:39349)
0808 15:43:06.432553 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:06.432559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:06.432580 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:43:06.432784 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:06.432789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:06.432858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:06.432994 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:06.433662 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.434528 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.435769 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.436191 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.436319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.440513 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:44519: BlockingRecv error: recv got EOF from 127.23.130.1:44519 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260808 15:43:06.441210 1641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:43:06.451246 1793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:06.451440 1793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:06.451743 1793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:43:06.476754 1793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/krb5kdc/oryx_127.23.130.3.keytab
--principal=oryx/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:36463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=35221
--tserver_master_addrs=127.23.130.62:39349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:06.479264 1793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:06.481673 1793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:06.500521 1801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:06.500937 1804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:06.501389 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:06.501884 1793 server_base.cc:1034] running on GCE node
I20260808 15:43:06.502985 1793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:06.504555 1793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:06.505944 1793 hybrid_clock.cc:648] HybridClock initialized: now 1786203786505855 us; error 55 us; skew 500 ppm
I20260808 15:43:06.506477 1793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203786, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:06.511595 1793 init.cc:377] Logged in from keytab as oryx/127.23.130.3@KRBTEST.COM (short username oryx)
I20260808 15:43:06.513242 1793 webserver.cc:460] Webserver started at http://127.23.130.3:35221/ using document root <none> and password file <none>
I20260808 15:43:06.513931 1793 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:06.514094 1793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:06.519907 1793 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:43:06.523061 1812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:06.524662 1793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:43:06.524833 1793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "a77af77c07054986b148f5e5522992f7"
format_stamp: "Formatted at 2026-08-08 15:43:03 on dist-test-slave-gp6n"
server_key: "55a388ed937c7ff5b389057128960d7c"
server_key_iv: "f3a9d691629ca152b31b9643d0da0639"
server_key_version: "encryptionkey@0"
I20260808 15:43:06.525789 1793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:06.548184 1793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:06.552587 1793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:06.552897 1793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:06.553972 1793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:06.555544 1793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:06.555667 1793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:06.555799 1793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:06.555884 1793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:06.595533 1793 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:36463
I20260808 15:43:06.595602 1936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:36463 every 8 connection(s)
I20260808 15:43:06.596990 1793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:43:06.603919 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 1793
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:06 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:06.626297 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:06.605127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.23.130.62:39349)
0808 15:43:06.611092 (+ 5965us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:06.611102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:06.611128 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:43:06.611232 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:06.611240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:06.611339 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:06.611490 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:06.613558 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.614657 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.616562 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.616960 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.617087 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.620084 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:06.620092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:06.620108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:06.620140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:06.623111 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:06.623772 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:06.623779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:06.623785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:06.623957 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:06.624359 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:06.624363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:06.624365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:06.624798 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:06.625132 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:06.625596 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 15:43:06.625980 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":5772,"thread_start_us":5561,"threads_started":1}
W20260808 15:43:07.018334 1792 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.245s user 0.005s sys 0.023s
W20260808 15:43:07.019815 1791 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:05.759087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:49257 (local address 127.23.130.62:39349)
0808 15:43:05.759558 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:05.759563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:05.761461 (+ 1898us) server_negotiation.cc:407] Connection header received
0808 15:43:05.764652 (+ 3191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:05.764662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:05.764789 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:05.764985 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:05.767133 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.768369 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.771011 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.772008 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.772197 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.019484 (+1247287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:49257: BlockingRecv error: recv got EOF from 127.23.130.2:49257 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
I20260808 15:43:07.020373 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:05.760159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:49257)
0808 15:43:05.761273 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:05.761333 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:05.764277 (+ 2944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:05.765222 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:05.765235 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:05.766025 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:05.766877 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:05.766905 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.768597 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.768605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:05.770806 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:05.770823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:05.772280 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:05.772289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:05.772504 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:05.773290 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:05.773336 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.019258 (+1245922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":478,"threads_started":1}
W20260808 15:43:07.021332 1789 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:39349 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.130.62:39349: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:07.037691 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:07.022156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:58325)
0808 15:43:07.022898 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:07.022914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:07.023013 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:07.023564 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:07.023572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:07.024044 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:07.024269 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.024278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.026279 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.026283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.027327 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.027338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.028845 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.028854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.029000 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.033442 (+ 4442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:07.033462 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.037484 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590}
W20260808 15:43:07.038231 1791 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:07.022566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:58325 (local address 127.23.130.62:39349)
0808 15:43:07.022830 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:07.022841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:07.022992 (+ 151us) server_negotiation.cc:407] Connection header received
0808 15:43:07.023125 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:07.023133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:07.023243 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:07.023376 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:07.024420 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.026107 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.027472 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.028558 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.028726 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.037886 (+ 9160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:58325: BlockingRecv error: recv got EOF from 127.23.130.2:58325 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:07.049728 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:07.038384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:46687)
0808 15:43:07.038741 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:07.038761 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:07.038866 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:07.039376 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:07.039389 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:07.039704 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:07.039861 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.039870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.041892 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.041896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.043015 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.043028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.044455 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.044460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.044622 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.045463 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:07.045493 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.049465 (+ 3972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239}
W20260808 15:43:07.050068 1789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:43:07.050292 1791 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:07.038610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:46687 (local address 127.23.130.62:39349)
0808 15:43:07.038914 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:07.038924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:07.038952 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:43:07.039021 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:07.039030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:07.039134 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:07.039279 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:07.040015 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.041698 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.043170 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.044239 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.044423 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.049832 (+ 5409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:46687: BlockingRecv error: recv got EOF from 127.23.130.2:46687 (error 108)
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":126}
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:07.455531 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:07.442010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:55001)
0808 15:43:07.442681 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:07.442700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:07.442839 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:07.443343 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:07.443349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:07.443813 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:07.444001 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.444011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.446194 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.446199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.447504 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.447519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.449079 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.449087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.449447 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.450486 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:07.450506 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.455220 (+ 4714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":422,"threads_started":1}
W20260808 15:43:07.456234 1791 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:07.442141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:55001 (local address 127.23.130.62:39349)
0808 15:43:07.442459 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:07.442470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:07.442794 (+ 324us) server_negotiation.cc:407] Connection header received
0808 15:43:07.442990 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:07.443000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:07.443150 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:07.443331 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:07.444164 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.445984 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.447682 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.448815 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.449029 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.455632 (+ 6603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:55001: BlockingRecv error: recv got EOF from 127.23.130.1:55001 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260808 15:43:07.767182 1940 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.151s user 0.007s sys 0.020s
W20260808 15:43:07.768311 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:06.600555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:43167 (local address 127.23.130.62:39349)
0808 15:43:06.600787 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:06.600794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:06.602162 (+ 1368us) server_negotiation.cc:407] Connection header received
0808 15:43:06.606080 (+ 3918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:06.606089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:06.606197 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:06.606315 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:06.609264 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.610770 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.613197 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.614378 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.614644 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.768093 (+1153449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:43167: BlockingRecv error: recv got EOF from 127.23.130.3:43167 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":85}
I20260808 15:43:07.768934 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:06.600888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:43167)
0808 15:43:06.601923 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:06.602002 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:06.605618 (+ 3616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:06.606734 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:06.606752 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:06.607808 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:06.608992 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.609030 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.610984 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.610991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.613028 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:06.613046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:06.614700 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:06.614711 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:06.614969 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:06.615788 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:06.615852 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.767965 (+1152113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":349,"threads_started":1}
W20260808 15:43:07.769855 1938 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:39349 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.130.62:39349: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:07.778826 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:07.770522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:50607)
0808 15:43:07.770704 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:07.770725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:07.770832 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:07.771127 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:07.771130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:07.771493 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:07.771656 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.771663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.772674 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.772677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.773644 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.773654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.774352 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.774355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.774555 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.775172 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:07.775188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.778627 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:43:07.779088 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:07.770624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:50607 (local address 127.23.130.62:39349)
0808 15:43:07.770886 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:07.770891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:07.770907 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:43:07.770952 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:07.770955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:07.771011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:07.771092 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:07.771782 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.772560 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.773768 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.774210 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.774306 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.778809 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:50607: BlockingRecv error: recv got EOF from 127.23.130.3:50607 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:07 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:07.787703 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:07.779513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:35759)
0808 15:43:07.779661 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:07.779678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:07.779757 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:07.780019 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:07.780022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:07.780291 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:07.780453 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.780461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.781569 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.781572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.782551 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:07.782562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.783219 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.783222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:07.783347 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.783861 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:07.783877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:07.787429 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:43:07.787962 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:07.779603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35759 (local address 127.23.130.62:39349)
0808 15:43:07.779776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:07.779780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:07.779796 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:43:07.779834 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:07.779837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:07.779900 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:07.779983 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:07.780603 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.781450 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.782695 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:07.783100 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:07.783201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:07.787687 (+ 4486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:35759: BlockingRecv error: recv got EOF from 127.23.130.3:35759 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 15:43:07.788002 1938 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:08.059671 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:08.050711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:49069)
0808 15:43:08.051240 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:08.051257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:08.051381 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:08.051714 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:08.051718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:08.052122 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:08.052269 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.052278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.053290 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.053294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.054486 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.054500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.055180 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.055190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.055321 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.055917 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:08.055934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:08.059417 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":328,"threads_started":1}
W20260808 15:43:08.059933 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:08.050784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:49069 (local address 127.23.130.62:39349)
0808 15:43:08.051021 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:08.051031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:08.051343 (+ 312us) server_negotiation.cc:407] Connection header received
0808 15:43:08.051504 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:08.051507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:08.051584 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:08.051667 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:08.052423 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.053144 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.054614 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.055021 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.055113 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.059658 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:49069: BlockingRecv error: recv got EOF from 127.23.130.2:49069 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:08.466558 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:08.456585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:42021)
0808 15:43:08.457116 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:08.457132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:08.457312 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:08.457658 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:08.457665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:08.458106 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:08.458293 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.458302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.459413 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.459419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.460343 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.460355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.461084 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.461091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.461257 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.461873 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:08.461894 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:08.466236 (+ 4342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":310,"threads_started":1}
W20260808 15:43:08.466769 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:08.456655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:42021 (local address 127.23.130.62:39349)
0808 15:43:08.456936 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:08.456945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:08.457223 (+ 278us) server_negotiation.cc:407] Connection header received
0808 15:43:08.457443 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:08.457448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:08.457517 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:08.457619 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:08.458520 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.459268 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.460482 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.460923 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.461047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.466510 (+ 5463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:42021: BlockingRecv error: recv got EOF from 127.23.130.1:42021 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:08 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:08.797919 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:08.788688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:49029)
0808 15:43:08.789389 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:08.789406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:08.789523 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:08.789836 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:08.789839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:08.790236 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:08.790460 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.790471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.791508 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.791512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.792471 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:08.792483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.793134 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.793138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:08.793259 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.793926 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:08.793945 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:08.797583 (+ 3638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":404,"threads_started":1}
W20260808 15:43:08.798032 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:08.788760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:49029 (local address 127.23.130.62:39349)
0808 15:43:08.788925 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:08.788931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:08.789478 (+ 547us) server_negotiation.cc:407] Connection header received
0808 15:43:08.789635 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:08.789639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:08.789712 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:08.789790 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:08.790602 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.791370 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.792590 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:08.792976 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:08.793064 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:08.797776 (+ 4712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:49029: BlockingRecv error: recv got EOF from 127.23.130.3:49029 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:09.069711 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:09.060893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:49651)
0808 15:43:09.061453 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:09.061470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:09.061639 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:09.061977 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:09.061982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:09.062500 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:09.062700 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.062709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.063731 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.063736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.064663 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.064674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.065371 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.065377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.065546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.066119 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:09.066136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:09.069476 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":316,"threads_started":1}
W20260808 15:43:09.069926 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:09.060974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:49651 (local address 127.23.130.62:39349)
0808 15:43:09.061231 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:09.061239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:09.061563 (+ 324us) server_negotiation.cc:407] Connection header received
0808 15:43:09.061767 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:09.061773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:09.061847 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:09.061945 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:09.062856 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.063588 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.064810 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.065209 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.065335 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.069679 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:49651: BlockingRecv error: recv got EOF from 127.23.130.2:49651 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:09.476903 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:09.467670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:37849)
0808 15:43:09.468219 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:09.468236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:09.468423 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:09.468776 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:09.468780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:09.469183 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:09.469439 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.469452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.470587 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.470591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.471546 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.471560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.472259 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.472264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.472446 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.473024 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:09.473042 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:09.476527 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":284,"threads_started":1}
W20260808 15:43:09.477027 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:09.467860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:37849 (local address 127.23.130.62:39349)
0808 15:43:09.468101 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:09.468109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:09.468314 (+ 205us) server_negotiation.cc:407] Connection header received
0808 15:43:09.468559 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:09.468564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:09.468635 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:09.468750 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:09.469606 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.470461 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.471689 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.472132 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.472226 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.476763 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:37849: BlockingRecv error: recv got EOF from 127.23.130.1:37849 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:09 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:09.808707 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:09.799060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:42151)
0808 15:43:09.799648 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:09.799666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:09.799789 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:09.800186 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:09.800197 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:09.800782 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:09.800978 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.800988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.802093 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.802103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.803087 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:09.803099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.803837 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.803844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:09.804020 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.804648 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:09.804667 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:09.808343 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":313,"threads_started":1}
W20260808 15:43:09.808833 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:09.799119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:42151 (local address 127.23.130.62:39349)
0808 15:43:09.799415 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:09.799422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:09.799764 (+ 342us) server_negotiation.cc:407] Connection header received
0808 15:43:09.799939 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:09.799944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:09.800036 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:09.800149 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:09.801123 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.801939 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.803226 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:09.803676 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:09.803804 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:09.808549 (+ 4745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:42151: BlockingRecv error: recv got EOF from 127.23.130.3:42151 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:10.080567 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:10.070792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:48383)
0808 15:43:10.071269 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:10.071285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:10.071440 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:10.071875 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:10.071892 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:10.072452 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:10.072657 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.072667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.073699 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.073702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.074998 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.075010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.075954 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.075959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.076093 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.076639 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:10.076656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:10.080320 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
W20260808 15:43:10.080869 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:10.070883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:48383 (local address 127.23.130.62:39349)
0808 15:43:10.071118 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:10.071125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:10.071369 (+ 244us) server_negotiation.cc:407] Connection header received
0808 15:43:10.071594 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:10.071600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:10.071672 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:10.071789 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:10.072796 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.073565 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.075153 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.075824 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.075918 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.080602 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:48383: BlockingRecv error: recv got EOF from 127.23.130.2:48383 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:10.487645 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:10.478054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:43729)
0808 15:43:10.478661 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:10.478678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:10.478846 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:10.479208 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:10.479213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:10.479636 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:10.479819 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.479828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.480845 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.480853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.481870 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.481882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.482712 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.482719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.482914 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.483721 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:10.483745 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:10.487386 (+ 3641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":362,"threads_started":1}
W20260808 15:43:10.487890 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:10.478152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:43729 (local address 127.23.130.62:39349)
0808 15:43:10.478504 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:10.478512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:10.478769 (+ 257us) server_negotiation.cc:407] Connection header received
0808 15:43:10.478987 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:10.478997 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:10.479056 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:10.479161 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:10.479970 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.480704 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.482016 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.482544 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.482671 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.487612 (+ 4941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:43729: BlockingRecv error: recv got EOF from 127.23.130.1:43729 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:10 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:10.820736 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:10.810005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:36657)
0808 15:43:10.810702 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:10.810719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:10.810911 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:10.811412 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:10.811418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:10.811841 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:10.812025 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.812059 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.813179 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.813187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.814208 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:10.814220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.815063 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.815069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:10.815347 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.816098 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:10.816123 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:10.820407 (+ 4284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":427,"threads_started":1}
W20260808 15:43:10.820801 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:10.810121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:36657 (local address 127.23.130.62:39349)
0808 15:43:10.810377 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:10.810385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:10.810828 (+ 443us) server_negotiation.cc:407] Connection header received
0808 15:43:10.811072 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:10.811083 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:10.811189 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:10.811311 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:10.812204 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.813033 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.814371 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:10.814900 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:10.815049 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:10.820590 (+ 5541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:36657: BlockingRecv error: recv got EOF from 127.23.130.3:36657 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:11.094710 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:11.081733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:46355)
0808 15:43:11.082746 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:11.082772 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:11.082951 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:11.083322 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:11.083327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:11.083951 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:11.084237 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.084255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.085416 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.085424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.087465 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.087486 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.088258 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.088266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.088512 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.089220 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:11.089254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:11.094327 (+ 5073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":712,"threads_started":1}
W20260808 15:43:11.094862 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:11.081979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:46355 (local address 127.23.130.62:39349)
0808 15:43:11.082220 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:11.082229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:11.082884 (+ 655us) server_negotiation.cc:407] Connection header received
0808 15:43:11.083084 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:11.083088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:11.083156 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:11.083243 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:11.084420 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.085261 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.087652 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.088097 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.088200 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.094592 (+ 6392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:46355: BlockingRecv error: recv got EOF from 127.23.130.2:46355 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:11.498593 1960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:11.488709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:49481)
0808 15:43:11.489291 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:11.489311 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:11.489445 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:11.489766 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:11.489769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:11.490445 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:11.490632 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.490669 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.491997 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.492001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.492990 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.493004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.493665 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.493669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.493800 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.494492 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:11.494511 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:11.498287 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
W20260808 15:43:11.498881 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:11.488811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49481 (local address 127.23.130.62:39349)
0808 15:43:11.489112 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:11.489118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:11.489401 (+ 283us) server_negotiation.cc:407] Connection header received
0808 15:43:11.489556 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:11.489560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:11.489623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:11.489733 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:11.490842 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.491844 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.493122 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.493539 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.493633 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.498589 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:49481: BlockingRecv error: recv got EOF from 127.23.130.1:49481 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:11 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:11.832087 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:11.821921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:37559)
0808 15:43:11.822553 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:11.822568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:11.822729 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:11.823118 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:11.823123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:11.823566 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:11.823736 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.823745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.824780 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.824785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.825847 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:11.825861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.826973 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.826981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:11.827129 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.827973 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:11.827998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:11.831814 (+ 3816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":322,"threads_started":1}
W20260808 15:43:11.832252 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:11.822077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:37559 (local address 127.23.130.62:39349)
0808 15:43:11.822314 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:11.822322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:11.822665 (+ 343us) server_negotiation.cc:407] Connection header received
0808 15:43:11.822871 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:11.822876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:11.822963 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:11.823075 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:11.823893 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.824637 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.826037 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:11.826797 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:11.826923 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:11.832017 (+ 5094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:37559: BlockingRecv error: recv got EOF from 127.23.130.3:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:12.110203 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:12.095726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:40311)
0808 15:43:12.096318 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:12.096336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:12.096503 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:12.096876 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:12.096880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:12.097363 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:12.097517 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.097526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.098615 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.098621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.099570 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.099582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.100292 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.100297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.100499 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.105922 (+ 5423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:12.105944 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:12.109852 (+ 3908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":341,"threads_started":1}
W20260808 15:43:12.110352 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:12.095887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:40311 (local address 127.23.130.62:39349)
0808 15:43:12.096130 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:12.096138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:12.096432 (+ 294us) server_negotiation.cc:407] Connection header received
0808 15:43:12.096638 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:12.096644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:12.096732 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:12.096846 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:12.097667 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.098449 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.099715 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.100145 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.100290 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.110078 (+ 9788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:40311: BlockingRecv error: recv got EOF from 127.23.130.2:40311 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:12.508589 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:12.499644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:59323)
0808 15:43:12.500141 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:12.500157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:12.500316 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:12.500692 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:12.500698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:12.501138 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:12.501308 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.501317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.502323 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.502330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.503287 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.503298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.504053 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.504059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.504247 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.504806 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:12.504823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:12.508216 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
W20260808 15:43:12.508695 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:12.499686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:59323 (local address 127.23.130.62:39349)
0808 15:43:12.499909 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:12.499917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:12.500256 (+ 339us) server_negotiation.cc:407] Connection header received
0808 15:43:12.500460 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:12.500465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:12.500552 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:12.500658 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:12.501461 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.502184 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.503439 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.503887 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.504016 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.508435 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:59323: BlockingRecv error: recv got EOF from 127.23.130.1:59323 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:12 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:12.842150 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:12.833194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:55687)
0808 15:43:12.833757 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:12.833774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:12.833905 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:12.834297 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:12.834301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:12.834784 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:12.834945 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.834954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.835942 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.835946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.836912 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:12.836924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.837622 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.837626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:12.837756 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.838450 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:12.838473 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:12.841919 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":313,"threads_started":1}
W20260808 15:43:12.842352 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:12.833264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:55687 (local address 127.23.130.62:39349)
0808 15:43:12.833547 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:12.833554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:12.833880 (+ 326us) server_negotiation.cc:407] Connection header received
0808 15:43:12.834017 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:12.834021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:12.834167 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:12.834256 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:12.835072 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.835806 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.837049 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:12.837494 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:12.837588 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:12.842115 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:55687: BlockingRecv error: recv got EOF from 127.23.130.3:55687 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:13.120815 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:13.111206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:45435)
0808 15:43:13.111813 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:13.111830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:13.111996 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:13.112380 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:13.112384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:13.112831 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:13.113006 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.113015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.114051 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.114057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.115127 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.115139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.115867 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.115873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.116061 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.116628 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:13.116644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:13.120570 (+ 3926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":351,"threads_started":1}
W20260808 15:43:13.121039 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:13.111269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:45435 (local address 127.23.130.62:39349)
0808 15:43:13.111500 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:13.111510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:13.111941 (+ 431us) server_negotiation.cc:407] Connection header received
0808 15:43:13.112152 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:13.112158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:13.112235 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:13.112336 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:13.113160 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.113909 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.115268 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.115699 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.115841 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.120782 (+ 4941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:45435: BlockingRecv error: recv got EOF from 127.23.130.2:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:13.519322 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:13.509671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:40919)
0808 15:43:13.510288 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:13.510304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:13.510475 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:13.510816 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:13.510820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:13.511242 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:13.511480 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.511489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.512645 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.512649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.513584 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.513596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.514792 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.514796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.514934 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.515616 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:13.515633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:13.519094 (+ 3461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":289,"threads_started":1}
W20260808 15:43:13.519598 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:13.509786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:40919 (local address 127.23.130.62:39349)
0808 15:43:13.509968 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:13.509973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:13.510383 (+ 410us) server_negotiation.cc:407] Connection header received
0808 15:43:13.510607 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:13.510612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:13.510680 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:13.510770 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:13.511609 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.512528 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.513768 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.514601 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.514758 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.519321 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:40919: BlockingRecv error: recv got EOF from 127.23.130.1:40919 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:13 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:13.852512 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:13.843266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:56745)
0808 15:43:13.843783 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:13.843799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:13.843915 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:13.844207 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:13.844211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:13.844628 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:13.844804 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.844812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.845858 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.845862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.847295 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:13.847325 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.848007 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.848011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:13.848135 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.848723 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:13.848741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:13.852269 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":292,"threads_started":1}
W20260808 15:43:13.852816 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:13.843369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:56745 (local address 127.23.130.62:39349)
0808 15:43:13.843540 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:13.843546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:13.843875 (+ 329us) server_negotiation.cc:407] Connection header received
0808 15:43:13.844007 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:13.844011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:13.844075 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:13.844163 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:13.844936 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.845721 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.847432 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:13.847870 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:13.847967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:13.852530 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:56745: BlockingRecv error: recv got EOF from 127.23.130.3:56745 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:14.131711 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:14.121844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:58619)
0808 15:43:14.122507 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:14.122524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:14.122648 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:14.123103 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:14.123108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:14.123648 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:14.123820 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.123830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.124864 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.124869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.125822 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.125833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.126610 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.126618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.126806 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.127372 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:14.127388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:14.131482 (+ 4094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
W20260808 15:43:14.131971 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:14.122005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:58619 (local address 127.23.130.62:39349)
0808 15:43:14.122180 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:14.122185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:14.122636 (+ 451us) server_negotiation.cc:407] Connection header received
0808 15:43:14.122814 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:14.122820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:14.122923 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:14.123055 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:14.123986 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.124702 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.125988 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.126379 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.126599 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.131686 (+ 5087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:58619: BlockingRecv error: recv got EOF from 127.23.130.2:58619 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:14.530385 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:14.520207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:51725)
0808 15:43:14.520821 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:14.520839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:14.520961 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:14.521257 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:14.521261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:14.521719 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:14.521919 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.521929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.523034 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.523038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.524078 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.524093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.524808 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.524813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.524939 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.525542 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:14.525559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:14.530001 (+ 4442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":330,"threads_started":1}
W20260808 15:43:14.530589 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:14.520307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:51725 (local address 127.23.130.62:39349)
0808 15:43:14.520486 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:14.520491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:14.520912 (+ 421us) server_negotiation.cc:407] Connection header received
0808 15:43:14.521059 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:14.521063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:14.521122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:14.521203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:14.522103 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.522878 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.524217 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.524635 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.524736 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.530260 (+ 5524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:51725: BlockingRecv error: recv got EOF from 127.23.130.1:51725 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:14 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:14.863606 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:14.853486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:43387)
0808 15:43:14.854193 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:14.854209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:14.854332 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:14.854881 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:14.854885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:14.855344 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:14.855544 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.855553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.856597 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.856601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.857594 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:14.857607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.858477 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.858491 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:14.858695 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.859304 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:14.859322 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:14.863249 (+ 3927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":329,"threads_started":1}
W20260808 15:43:14.863732 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:14.853696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:43387 (local address 127.23.130.62:39349)
0808 15:43:14.853879 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:14.853883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:14.854290 (+ 407us) server_negotiation.cc:407] Connection header received
0808 15:43:14.854646 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:14.854650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:14.854714 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:14.854808 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:14.855689 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.856434 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.857751 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:14.858254 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:14.858393 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:14.863442 (+ 5049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:43387: BlockingRecv error: recv got EOF from 127.23.130.3:43387 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:15.148222 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:15.132677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:39805)
0808 15:43:15.133314 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:15.133333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:15.133459 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:15.133893 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:15.133897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:15.134322 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:15.134519 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.134528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.135953 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.135957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.137209 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.137228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.138215 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.138220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.138339 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.144049 (+ 5710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:15.144068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:15.147953 (+ 3885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":339,"threads_started":1}
W20260808 15:43:15.148713 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:15.132757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:39805 (local address 127.23.130.62:39349)
0808 15:43:15.132924 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:15.132929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:15.133407 (+ 478us) server_negotiation.cc:407] Connection header received
0808 15:43:15.133564 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:15.133567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:15.133654 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:15.133785 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:15.134733 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.135788 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.137380 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.138019 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.138118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.148348 (+ 10230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:39805: BlockingRecv error: recv got EOF from 127.23.130.2:39805 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:15.541047 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:15.531638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:33575)
0808 15:43:15.532132 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:15.532149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:15.532259 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:15.532645 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:15.532650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:15.533079 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:15.533224 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.533232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.534581 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.534587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.535804 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.535818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.536453 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.536457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.536575 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.537167 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:15.537184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:15.540816 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":298,"threads_started":1}
W20260808 15:43:15.541219 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:15.531665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:33575 (local address 127.23.130.62:39349)
0808 15:43:15.531881 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:15.531891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:15.532226 (+ 335us) server_negotiation.cc:407] Connection header received
0808 15:43:15.532369 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:15.532373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:15.532434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:15.532546 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:15.533341 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.534380 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.535932 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.536323 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.536417 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.541004 (+ 4587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:33575: BlockingRecv error: recv got EOF from 127.23.130.1:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:15 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:15.873778 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:15.864592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:54265)
0808 15:43:15.865173 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:15.865189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:15.865304 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:15.865604 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:15.865608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:15.866002 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:15.866175 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.866183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.867317 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.867324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.868595 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:15.868608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.869377 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.869382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:15.869503 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.870107 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:15.870125 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:15.873526 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":318,"threads_started":1}
W20260808 15:43:15.873917 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:15.864687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:54265 (local address 127.23.130.62:39349)
0808 15:43:15.864850 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:15.864854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:15.865263 (+ 409us) server_negotiation.cc:407] Connection header received
0808 15:43:15.865402 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:15.865405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:15.865469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:15.865555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:15.866293 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.867183 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.868723 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:15.869180 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:15.869286 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:15.873723 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:54265: BlockingRecv error: recv got EOF from 127.23.130.3:54265 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:16.159158 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:16.149306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:57533)
0808 15:43:16.149851 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:16.149868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:16.149989 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:16.150312 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:16.150315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:16.150802 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:16.150965 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.150973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.152106 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.152110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.153058 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.153068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.153938 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.153942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.154148 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.154975 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:16.154993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:16.158930 (+ 3937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":305,"threads_started":1}
W20260808 15:43:16.159482 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:16.149411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:57533 (local address 127.23.130.62:39349)
0808 15:43:16.149609 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:16.149614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:16.149948 (+ 334us) server_negotiation.cc:407] Connection header received
0808 15:43:16.150108 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:16.150112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:16.150177 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:16.150271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:16.151098 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.151979 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.153212 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.153762 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.153930 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.159157 (+ 5227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:57533: BlockingRecv error: recv got EOF from 127.23.130.2:57533 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:16.551631 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:16.542156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:35695)
0808 15:43:16.542753 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:16.542770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:16.542887 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:16.543206 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:16.543210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:16.543596 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:16.543762 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.543770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.544824 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.544827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.545810 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.545822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.546760 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.546765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.546888 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.547583 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:16.547601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:16.551283 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":332,"threads_started":1}
W20260808 15:43:16.551853 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:16.542200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:35695 (local address 127.23.130.62:39349)
0808 15:43:16.542470 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:16.542481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:16.542856 (+ 375us) server_negotiation.cc:407] Connection header received
0808 15:43:16.542994 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:16.542998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:16.543064 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:16.543154 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:16.543896 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.544691 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.545968 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.546586 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.546684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.551492 (+ 4808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:35695: BlockingRecv error: recv got EOF from 127.23.130.1:35695 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:16 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:16.884341 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:16.874991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:44201)
0808 15:43:16.875662 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:16.875679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:16.875793 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:16.876084 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:16.876087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:16.876552 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:16.876696 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.876705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.877763 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.877768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.878888 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:16.878902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.879600 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.879604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:16.879734 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.880301 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:16.880319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:16.884113 (+ 3794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":401,"threads_started":1}
W20260808 15:43:16.884583 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:16.875049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:44201 (local address 127.23.130.62:39349)
0808 15:43:16.875285 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:16.875293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:16.875781 (+ 488us) server_negotiation.cc:407] Connection header received
0808 15:43:16.875892 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:16.875895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:16.875957 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:16.876041 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:16.876849 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.877619 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.879060 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:16.879463 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:16.879566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:16.884304 (+ 4738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:44201: BlockingRecv error: recv got EOF from 127.23.130.3:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:17.174612 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:17.160054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:37231)
0808 15:43:17.160574 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:17.160590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:17.160751 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:17.161112 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:17.161117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:17.161517 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:17.161692 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.161701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.162900 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.162909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.163900 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.163913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.164651 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.164656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.164831 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.170695 (+ 5864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:17.170713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:17.174291 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
W20260808 15:43:17.174750 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:17.160122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:37231 (local address 127.23.130.62:39349)
0808 15:43:17.160403 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:17.160409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:17.160681 (+ 272us) server_negotiation.cc:407] Connection header received
0808 15:43:17.160904 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:17.160910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:17.160983 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:17.161082 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:17.161842 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.162766 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.164045 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.164493 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.164622 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.174551 (+ 9929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:37231: BlockingRecv error: recv got EOF from 127.23.130.2:37231 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:17.563218 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:17.552682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:59225)
0808 15:43:17.553330 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:17.553349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:17.553473 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:17.554024 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:17.554028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:17.554619 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:17.554833 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.554843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.555929 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.555932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.556946 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.556957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.557706 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.557711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.557856 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.558549 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:17.558568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:17.562839 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":391,"threads_started":1}
W20260808 15:43:17.563328 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:17.552955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:59225 (local address 127.23.130.62:39349)
0808 15:43:17.553216 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:17.553221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:17.553430 (+ 209us) server_negotiation.cc:407] Connection header received
0808 15:43:17.553713 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:17.553718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:17.553823 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:17.553961 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:17.554970 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.555799 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.557093 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.557562 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.557671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.563042 (+ 5371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:59225: BlockingRecv error: recv got EOF from 127.23.130.1:59225 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:17 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:17.894917 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:17.885349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:53981)
0808 15:43:17.885905 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:17.885922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:17.886077 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:17.886479 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:17.886487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:17.886914 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:17.887077 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.887086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.888118 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.888124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.889121 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:17.889132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.889863 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.889869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:17.890050 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.890744 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:17.890764 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:17.894582 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
W20260808 15:43:17.895012 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:17.885485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:53981 (local address 127.23.130.62:39349)
0808 15:43:17.885715 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:17.885723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:17.886007 (+ 284us) server_negotiation.cc:407] Connection header received
0808 15:43:17.886200 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:17.886205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:17.886276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:17.886377 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:17.887227 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.887980 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.889265 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:17.889705 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:17.889878 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:17.894755 (+ 4877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:53981: BlockingRecv error: recv got EOF from 127.23.130.3:53981 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:18.185355 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:18.175672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:34741)
0808 15:43:18.176280 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:18.176298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:18.176462 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:18.176767 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:18.176770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:18.177191 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:18.177415 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.177425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.178537 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.178540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.179453 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.179463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.180158 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.180163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.180298 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.180890 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:18.180906 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:18.185018 (+ 4112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":365,"threads_started":1}
W20260808 15:43:18.185494 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:18.175900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:34741 (local address 127.23.130.62:39349)
0808 15:43:18.176140 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:18.176147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:18.176426 (+ 279us) server_negotiation.cc:407] Connection header received
0808 15:43:18.176566 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:18.176569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:18.176631 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:18.176712 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:18.177570 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.178371 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.179579 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.180018 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.180112 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.185228 (+ 5116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:34741: BlockingRecv error: recv got EOF from 127.23.130.2:34741 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:18.573678 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:18.564278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:38533)
0808 15:43:18.564862 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:18.564879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:18.565060 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:18.565405 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:18.565410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:18.565888 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:18.566066 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.566075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.567173 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.567178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.568140 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.568152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.568888 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.568902 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.569093 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.569730 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:18.569748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:18.573382 (+ 3634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":333,"threads_started":1}
W20260808 15:43:18.573912 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:18.564356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:38533 (local address 127.23.130.62:39349)
0808 15:43:18.564601 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:18.564610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:18.564989 (+ 379us) server_negotiation.cc:407] Connection header received
0808 15:43:18.565185 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:18.565190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:18.565265 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:18.565367 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:18.566213 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.567022 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.568283 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.568729 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.568862 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.573625 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:38533: BlockingRecv error: recv got EOF from 127.23.130.1:38533 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:18 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:18.905292 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:18.896055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:50687)
0808 15:43:18.896615 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:18.896631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:18.896791 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:18.897142 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:18.897146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:18.897612 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:18.897831 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.897840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.898923 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.898928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.899867 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:18.899879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.900605 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.900610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:18.900793 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.901454 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:18.901471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:18.905060 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":335,"threads_started":1}
W20260808 15:43:18.905678 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:18.896086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:50687 (local address 127.23.130.62:39349)
0808 15:43:18.896263 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:18.896273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:18.896716 (+ 443us) server_negotiation.cc:407] Connection header received
0808 15:43:18.896928 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:18.896933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:18.897001 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:18.897104 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:18.897977 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.898780 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.900015 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:18.900453 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:18.900590 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:18.905414 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:50687: BlockingRecv error: recv got EOF from 127.23.130.3:50687 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:19.195320 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:19.186623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:43747)
0808 15:43:19.187187 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:19.187204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:19.187319 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:19.187636 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:19.187639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:19.188024 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:19.188197 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.188205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.189151 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.189154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.190120 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.190133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.190944 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.190949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.191083 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.191657 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:19.191673 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:19.195097 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":311,"threads_started":1}
W20260808 15:43:19.195509 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:19.186638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:43747 (local address 127.23.130.62:39349)
0808 15:43:19.186867 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:19.186872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:19.187284 (+ 412us) server_negotiation.cc:407] Connection header received
0808 15:43:19.187431 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:19.187434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:19.187499 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:19.187584 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:19.188320 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.189025 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.190242 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.190781 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.190875 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.195268 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:43747: BlockingRecv error: recv got EOF from 127.23.130.2:43747 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:19.584671 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:19.575080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:54077)
0808 15:43:19.575590 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:19.575605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:19.575762 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:19.576113 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:19.576118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:19.576560 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:19.576716 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.576725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.577807 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.577813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.578799 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.578812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.579601 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.579607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.579783 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.580511 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:19.580536 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:19.584441 (+ 3905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
W20260808 15:43:19.584959 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:19.575149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:54077 (local address 127.23.130.62:39349)
0808 15:43:19.575416 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:19.575424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:19.575691 (+ 267us) server_negotiation.cc:407] Connection header received
0808 15:43:19.575885 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:19.575892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:19.575970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:19.576076 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:19.576886 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.577660 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.578942 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.579454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.579593 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.584639 (+ 5046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:54077: BlockingRecv error: recv got EOF from 127.23.130.1:54077 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:19 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:19.915797 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:19.906456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:43291)
0808 15:43:19.907065 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:19.907080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:19.907228 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:19.907575 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:19.907580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:19.908031 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:19.908206 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.908214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.909257 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.909264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.910240 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:19.910251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.911071 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.911078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:19.911261 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.911858 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:19.911875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:19.915442 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":305,"threads_started":1}
W20260808 15:43:19.915870 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:19.906496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:43291 (local address 127.23.130.62:39349)
0808 15:43:19.906699 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:19.906707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:19.907164 (+ 457us) server_negotiation.cc:407] Connection header received
0808 15:43:19.907356 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:19.907361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:19.907442 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:19.907545 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:19.908348 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.909120 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.910424 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:19.910875 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:19.911030 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:19.915644 (+ 4614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:43291: BlockingRecv error: recv got EOF from 127.23.130.3:43291 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:20.205689 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:20.196288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:34759)
0808 15:43:20.196844 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:20.196860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:20.197017 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:20.197361 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:20.197366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:20.197840 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:20.197997 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.198005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.199090 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.199095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.200016 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.200027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.200779 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.200784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.200970 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.201612 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:20.201630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:20.205320 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":322,"threads_started":1}
W20260808 15:43:20.205822 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:20.196495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:34759 (local address 127.23.130.62:39349)
0808 15:43:20.196754 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:20.196760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:20.196947 (+ 187us) server_negotiation.cc:407] Connection header received
0808 15:43:20.197155 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:20.197162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:20.197226 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:20.197330 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:20.198145 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.198934 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.200162 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.200631 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.200769 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.205517 (+ 4748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:34759: BlockingRecv error: recv got EOF from 127.23.130.2:34759 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:20.595221 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:20.585889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:45513)
0808 15:43:20.586448 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:20.586470 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:20.586585 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:20.586877 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:20.586880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:20.587295 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:20.587496 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.587505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.588567 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.588571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.589505 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.589516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.590184 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.590188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.590305 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.591013 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:20.591031 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:20.594856 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":293,"threads_started":1}
W20260808 15:43:20.595312 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:20.585985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:45513 (local address 127.23.130.62:39349)
0808 15:43:20.586192 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:20.586197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:20.586548 (+ 351us) server_negotiation.cc:407] Connection header received
0808 15:43:20.586686 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:20.586689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:20.586752 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:20.586841 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:20.587655 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.588433 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.589631 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.590051 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.590144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.595053 (+ 4909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:45513: BlockingRecv error: recv got EOF from 127.23.130.1:45513 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:20 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.3@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:20.926024 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:20.916931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.3:35855)
0808 15:43:20.917455 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:20.917474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:20.917615 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:20.917945 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:20.917962 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:20.918372 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:20.918561 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.918569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.919614 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.919618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.920569 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:20.920581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.921260 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.921265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:20.921395 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.921979 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:20.921996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:20.925658 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
W20260808 15:43:20.926127 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:20.917054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35855 (local address 127.23.130.62:39349)
0808 15:43:20.917276 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:20.917281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:20.917547 (+ 266us) server_negotiation.cc:407] Connection header received
0808 15:43:20.917747 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:20.917751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:20.917810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:20.917885 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:20.918679 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.919470 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.920707 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:20.921121 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:20.921218 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:20.925879 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.3:35855: BlockingRecv error: recv got EOF from 127.23.130.3:35855 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 08 15:43:21 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:21 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.2@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:21.217264 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:21.207222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.2:34713)
0808 15:43:21.207786 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:21.207801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:21.207959 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:21.208328 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:21.208335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:21.208754 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:21.208935 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:21.208944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.210020 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.210025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:21.211104 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:21.211125 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.212217 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.212223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:21.212421 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.213134 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:21.213152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:21.216870 (+ 3718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":315,"threads_started":1}
W20260808 15:43:21.217324 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:21.207353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:34713 (local address 127.23.130.62:39349)
0808 15:43:21.207618 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:21.207624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:21.207889 (+ 265us) server_negotiation.cc:407] Connection header received
0808 15:43:21.208103 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:21.208109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:21.208183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:21.208287 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:21.209089 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.209865 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.211319 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.212047 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.212194 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.217091 (+ 4897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.2:34713: BlockingRecv error: recv got EOF from 127.23.130.2:34713 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 15:43:21 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:43:21 dist-test-slave-gp6n krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.130.1@KRBTEST.COM for oryx/127.23.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:43:21.605921 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:21.596506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39349 (local address 127.23.130.1:44815)
0808 15:43:21.597098 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:21.597115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:21.597228 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:21.597576 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:21.597579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:21.597983 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:21.598152 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:21.598160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.599638 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.599641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:21.600578 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:21.600590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.601281 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.601285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:21.601399 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.602053 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:21.602072 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:21.605578 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.130.62:39349: Server oryx/127.23.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":335,"threads_started":1}
W20260808 15:43:21.606053 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:21.596520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:44815 (local address 127.23.130.62:39349)
0808 15:43:21.596728 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:21.596734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:21.597213 (+ 479us) server_negotiation.cc:407] Connection header received
0808 15:43:21.597348 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:21.597354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:21.597434 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:21.597537 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:21.598298 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.599504 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.600714 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.601148 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.601245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.605761 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.130.1:44815: BlockingRecv error: recv got EOF from 127.23.130.1:44815 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:43:21.606775 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1476
I20260808 15:43:21.620184 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:21.608023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.23.130.1:46209)
0808 15:43:21.608778 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:21.608785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:21.609225 (+ 440us) server_negotiation.cc:407] Connection header received
0808 15:43:21.609423 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:21.609429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:21.609660 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:21.609809 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:21.611360 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.612555 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.613713 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.614260 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.614406 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.616990 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:21.616998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:21.617011 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:21.617056 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:21.619704 (+ 2648us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:43:21.619966 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47880: Request ticket server kudu/127.23.130.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":429,"threads_started":1}
W20260808 15:43:21.620383 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1476: Not authorized: Client connection negotiation failed: client connection to 127.23.130.1:46209: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:43:21.620620 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1476
I20260808 15:43:21.641690 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1645
I20260808 15:43:21.654987 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:21.643024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.23.130.2:41535)
0808 15:43:21.643751 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:21.643758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:21.643788 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:43:21.643859 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:21.643866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:21.644109 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:21.644377 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:21.645770 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.646956 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.648238 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.648836 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.648987 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.651703 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:21.651719 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:21.651735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:21.651778 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:21.654234 (+ 2456us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:43:21.654649 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38670: Request ticket server kudu/127.23.130.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":415,"threads_started":1}
W20260808 15:43:21.655100 1996 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38670: Request ticket server kudu/127.23.130.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:43:21.655261 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1645: Not authorized: Client connection negotiation failed: client connection to 127.23.130.2:41535: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:43:21.655459 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1645
I20260808 15:43:21.674899 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1793
I20260808 15:43:21.687745 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:21.675929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.23.130.3:36463)
0808 15:43:21.676541 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:21.676550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:21.676582 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:43:21.676658 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:21.676667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:21.676985 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:21.677201 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:21.678522 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.680029 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.681173 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:21.681704 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:21.681803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:21.684364 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:21.684374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:21.684387 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:21.684443 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:21.686944 (+ 2501us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:43:21.687458 (+ 514us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55104: Request ticket server kudu/127.23.130.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":277,"threads_started":1}
W20260808 15:43:21.687793 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1793: Not authorized: Client connection negotiation failed: client connection to 127.23.130.3:36463: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.130.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:43:21.687863 1997 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55104: Request ticket server kudu/127.23.130.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:43:21.687950 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1793
I20260808 15:43:21.704756 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 1397
I20260808 15:43:21.706158 1469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:21.823421 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 1397
2026-08-08T15:43:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21955 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T15:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:43:21Z Disabled control of system clock
I20260808 15:43:21.911995 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45921
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:45921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:43:22.254835 2003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:22.255139 2003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:22.255221 2003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:22.262471 2003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:43:22.262583 2003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:22.262622 2003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:22.262672 2003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:43:22.262717 2003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:43:22.271920 2003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:45921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:22.273185 2003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:22.275081 2003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:22.287541 2008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:22.287595 2013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:22.287483 2009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:22.287730 2003 server_base.cc:1034] running on GCE node
I20260808 15:43:22.288901 2003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:22.290315 2003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:22.291549 2003 hybrid_clock.cc:648] HybridClock initialized: now 1786203802291493 us; error 48 us; skew 500 ppm
I20260808 15:43:22.291893 2003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:22.298720 2003 webserver.cc:460] Webserver started at http://127.23.130.62:34173/ using document root <none> and password file <none>
I20260808 15:43:22.299431 2003 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:22.299579 2003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:22.299913 2003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:22.302870 2003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "f4d13ac49ca6460697dd19194aed1ccf"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "befb8746fe0f46a89087c0d4a1f3e96d"
server_key_iv: "c4fe27ff8d5b211a3aebc2ae875071c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.303681 2003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "f4d13ac49ca6460697dd19194aed1ccf"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "befb8746fe0f46a89087c0d4a1f3e96d"
server_key_iv: "c4fe27ff8d5b211a3aebc2ae875071c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.308843 2003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:43:22.312335 2020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:22.313689 2003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:43:22.313874 2003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "f4d13ac49ca6460697dd19194aed1ccf"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "befb8746fe0f46a89087c0d4a1f3e96d"
server_key_iv: "c4fe27ff8d5b211a3aebc2ae875071c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.314091 2003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:22.349917 2003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:22.350888 2003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:22.351198 2003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:22.370543 2003 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:45921
I20260808 15:43:22.370632 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:45921 every 8 connection(s)
I20260808 15:43:22.371977 2003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:43:22.376266 2078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:22.376784 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2003
I20260808 15:43:22.377023 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:43:22.377713 24072 external_mini_cluster.cc:1428] Setting key 94d1ad6cd4256c82baadeafe8bd9c347
I20260808 15:43:22.389953 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf: Bootstrap starting.
I20260808 15:43:22.393959 2078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:22.395372 2078 log.cc:824] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:22.396314 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:22.380075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.23.130.62:45921)
0808 15:43:22.381358 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:22.381373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:22.383094 (+ 1721us) server_negotiation.cc:407] Connection header received
0808 15:43:22.385395 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:22.385422 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:22.386069 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:22.386550 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:22.387108 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.389072 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.390578 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.391777 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.391985 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:22.393362 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:22.393374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:22.393391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:22.394152 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:22.394296 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:22.394326 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:22.394498 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:22.394720 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 15:43:22.395049 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":450,"threads_started":1}
I20260808 15:43:22.398331 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf: No bootstrap required, opened a new log
I20260808 15:43:22.405347 2078 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } }
I20260808 15:43:22.405701 2078 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:22.405825 2078 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d13ac49ca6460697dd19194aed1ccf, State: Initialized, Role: FOLLOWER
I20260808 15:43:22.406368 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } }
I20260808 15:43:22.406652 2078 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:43:22.406787 2078 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:43:22.406957 2078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:22.409158 2078 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } }
I20260808 15:43:22.409675 2078 leader_election.cc:302] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4d13ac49ca6460697dd19194aed1ccf; no voters:
I20260808 15:43:22.410225 2078 leader_election.cc:288] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:43:22.410367 2083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:43:22.411190 2083 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [term 1 LEADER]: Becoming Leader. State: Replica: f4d13ac49ca6460697dd19194aed1ccf, State: Running, Role: LEADER
I20260808 15:43:22.411759 2083 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } }
I20260808 15:43:22.412331 2078 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:43:22.414659 2085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4d13ac49ca6460697dd19194aed1ccf. Latest consensus state: current_term: 1 leader_uuid: "f4d13ac49ca6460697dd19194aed1ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } } }
I20260808 15:43:22.415124 2085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:22.416440 2084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4d13ac49ca6460697dd19194aed1ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d13ac49ca6460697dd19194aed1ccf" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45921 } } }
I20260808 15:43:22.416864 2084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:22.418048 2090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:43:22.423207 2090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:43:22.431888 2090 catalog_manager.cc:1347] Generated new cluster ID: 7b29aee3bee3471491c94553f61c514c
I20260808 15:43:22.432039 2090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:43:22.452426 2090 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:43:22.453716 2090 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:43:22.471374 2090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4d13ac49ca6460697dd19194aed1ccf: Generated new TSK 0
I20260808 15:43:22.472236 2090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:43:22.482262 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--builtin_ntp_servers=127.23.130.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 15:43:22.828361 2102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:22.828644 2102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:22.828729 2102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:22.835887 2102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:22.835990 2102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:22.836097 2102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:43:22.844933 2102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:22.846179 2102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:22.847987 2102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:22.861492 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:22.861545 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:22.862246 2112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:22.862362 2102 server_base.cc:1034] running on GCE node
I20260808 15:43:22.862978 2102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:22.863858 2102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:22.865058 2102 hybrid_clock.cc:648] HybridClock initialized: now 1786203802865019 us; error 30 us; skew 500 ppm
I20260808 15:43:22.865339 2102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:22.867544 2102 webserver.cc:460] Webserver started at http://127.23.130.1:34801/ using document root <none> and password file <none>
I20260808 15:43:22.868198 2102 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:22.868284 2102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:22.868556 2102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:22.871470 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "335ed0dc860f4110a05987aae55dcc58"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "016da89265a94db059524046c3ae8640"
server_key_iv: "8ecacabbf41dd62b966cc59fc3593348"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.872280 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "335ed0dc860f4110a05987aae55dcc58"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "016da89265a94db059524046c3ae8640"
server_key_iv: "8ecacabbf41dd62b966cc59fc3593348"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.877373 2102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:43:22.880812 2119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:22.882154 2102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 15:43:22.882333 2102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "335ed0dc860f4110a05987aae55dcc58"
format_stamp: "Formatted at 2026-08-08 15:43:22 on dist-test-slave-gp6n"
server_key: "016da89265a94db059524046c3ae8640"
server_key_iv: "8ecacabbf41dd62b966cc59fc3593348"
server_key_version: "encryptionkey@0"
I20260808 15:43:22.882599 2102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:22.903937 2102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:22.904750 2102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:22.905067 2102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:22.905992 2102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:22.907488 2102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:22.907617 2102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:22.907740 2102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:22.907829 2102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:22.942895 2102 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:37931
I20260808 15:43:22.942993 2243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:37931 every 8 connection(s)
I20260808 15:43:22.944236 2102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:43:22.945065 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2102
I20260808 15:43:22.945297 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:43:22.946192 24072 external_mini_cluster.cc:1428] Setting key 2b4782b84f83679a73786a6ce984ac6a
I20260808 15:43:22.949433 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--builtin_ntp_servers=127.23.130.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:43:22.958770 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:22.947720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:55295 (local address 127.23.130.62:45921)
0808 15:43:22.948203 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:22.948211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:22.949787 (+ 1576us) server_negotiation.cc:407] Connection header received
0808 15:43:22.951636 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:22.951645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:22.951737 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:22.951867 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:22.952896 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.954013 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.955625 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.956336 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.956478 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:22.957768 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:22.957776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:22.957783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:22.957809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:22.957872 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:22.957891 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:22.957988 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:22.958184 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 15:43:22.958290 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260808 15:43:22.959542 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:22.948494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45921 (local address 127.23.130.1:55295)
0808 15:43:22.949622 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:22.949668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:22.951362 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:22.952057 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:22.952065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:22.952151 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:22.952720 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:22.952738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.954164 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.954168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:22.955482 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:22.955494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:22.956541 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:22.956546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:22.956750 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:22.957463 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:22.957466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:22.957505 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:22.957582 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:22.957991 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:22.957998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:22.958003 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:43:22.958139 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 15:43:22.958491 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":361,"threads_started":1}
I20260808 15:43:22.961263 2244 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45921
I20260808 15:43:22.961576 2244 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:22.962352 2244 heartbeater.cc:511] Master 127.23.130.62:45921 requested a full tablet report, sending...
I20260808 15:43:22.964663 2039 ts_manager.cc:194] Registered new tserver with Master: 335ed0dc860f4110a05987aae55dcc58 (127.23.130.1:37931)
I20260808 15:43:22.966979 2039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.1:55295
W20260808 15:43:23.289199 2248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:23.289513 2248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:23.289603 2248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:23.296794 2248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:23.296896 2248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:23.297003 2248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:43:23.307065 2248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:23.308320 2248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:23.310057 2248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:23.323489 2254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:23.323472 2257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:23.323707 2255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:23.324538 2248 server_base.cc:1034] running on GCE node
I20260808 15:43:23.325104 2248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:23.326015 2248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:23.327220 2248 hybrid_clock.cc:648] HybridClock initialized: now 1786203803327146 us; error 64 us; skew 500 ppm
I20260808 15:43:23.327503 2248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:23.329753 2248 webserver.cc:460] Webserver started at http://127.23.130.2:35479/ using document root <none> and password file <none>
I20260808 15:43:23.330502 2248 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:23.330614 2248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:23.330965 2248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:23.333797 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "6d0321c716f740be8615e51ea5743b1a"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "07532acceb3c9121ebcab6dcd474fac8"
server_key_iv: "25d7c10828468672e356913808d21d6d"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.334730 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "6d0321c716f740be8615e51ea5743b1a"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "07532acceb3c9121ebcab6dcd474fac8"
server_key_iv: "25d7c10828468672e356913808d21d6d"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.339883 2248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:43:23.343320 2263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.344564 2248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:43:23.344741 2248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "6d0321c716f740be8615e51ea5743b1a"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "07532acceb3c9121ebcab6dcd474fac8"
server_key_iv: "25d7c10828468672e356913808d21d6d"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.344955 2248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:23.371536 2248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:23.372412 2248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:23.372730 2248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:23.373751 2248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:23.375294 2248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:23.375427 2248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.375553 2248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:23.375641 2248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.411792 2248 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:35091
I20260808 15:43:23.411885 2386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:35091 every 8 connection(s)
I20260808 15:43:23.413259 2248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:43:23.421322 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2248
I20260808 15:43:23.421550 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:43:23.422076 24072 external_mini_cluster.cc:1428] Setting key 2d7900e6c116bb0bc1e09cf6fe5ed0e2
I20260808 15:43:23.425120 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--builtin_ntp_servers=127.23.130.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:43:23.426759 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:23.415929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:51003 (local address 127.23.130.62:45921)
0808 15:43:23.416117 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:23.416121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:23.417594 (+ 1473us) server_negotiation.cc:407] Connection header received
0808 15:43:23.419637 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:23.419641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:23.419707 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:23.419799 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:23.421087 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.421928 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.423919 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.424378 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.424530 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:23.425896 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:23.425902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:23.425907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:23.425933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:23.425967 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:23.425979 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:23.426075 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:23.426300 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 15:43:23.426396 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:43:23.427750 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:23.416474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45921 (local address 127.23.130.2:51003)
0808 15:43:23.417390 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:23.417479 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:23.419385 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:23.420095 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:23.420104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:23.420208 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:23.420899 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:23.420928 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.422093 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.422103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:23.423709 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:23.423723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.424620 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.424627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:23.424854 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:23.425580 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:23.425582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:23.425620 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:23.425698 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:23.426076 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:23.426090 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:23.426100 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:43:23.426287 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 15:43:23.426706 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
I20260808 15:43:23.429147 2387 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45921
I20260808 15:43:23.429508 2387 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:23.430544 2387 heartbeater.cc:511] Master 127.23.130.62:45921 requested a full tablet report, sending...
I20260808 15:43:23.432278 2039 ts_manager.cc:194] Registered new tserver with Master: 6d0321c716f740be8615e51ea5743b1a (127.23.130.2:35091)
I20260808 15:43:23.433624 2039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.2:51003
W20260808 15:43:23.759907 2391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:23.760236 2391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:23.760411 2391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:23.767500 2391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:23.767645 2391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:23.767860 2391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:43:23.776837 2391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:23.778127 2391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:23.779978 2391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:23.793421 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:23.793469 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:23.793519 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:23.794582 2391 server_base.cc:1034] running on GCE node
I20260808 15:43:23.795199 2391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:23.796120 2391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:23.797318 2391 hybrid_clock.cc:648] HybridClock initialized: now 1786203803797267 us; error 51 us; skew 500 ppm
I20260808 15:43:23.797634 2391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:23.800014 2391 webserver.cc:460] Webserver started at http://127.23.130.3:32851/ using document root <none> and password file <none>
I20260808 15:43:23.800727 2391 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:23.800863 2391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:23.801222 2391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:23.804141 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "944494a0eed0425b93af9dabfe62496b"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "9927a078f2cdc4e0839a6ff4dbaa7c80"
server_key_iv: "0cd9d41f73fd3d552f2812851f0b1143"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.804965 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "944494a0eed0425b93af9dabfe62496b"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "9927a078f2cdc4e0839a6ff4dbaa7c80"
server_key_iv: "0cd9d41f73fd3d552f2812851f0b1143"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.810226 2391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:43:23.813788 2407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.815137 2391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:23.815315 2391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "944494a0eed0425b93af9dabfe62496b"
format_stamp: "Formatted at 2026-08-08 15:43:23 on dist-test-slave-gp6n"
server_key: "9927a078f2cdc4e0839a6ff4dbaa7c80"
server_key_iv: "0cd9d41f73fd3d552f2812851f0b1143"
server_key_version: "encryptionkey@0"
I20260808 15:43:23.815542 2391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:23.839784 2391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:23.840649 2391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:23.840977 2391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:23.841943 2391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:23.843477 2391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:23.843602 2391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.843737 2391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:23.843833 2391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:23.877707 2391 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:44697
I20260808 15:43:23.877799 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:44697 every 8 connection(s)
I20260808 15:43:23.879133 2391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:43:23.884544 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2391
I20260808 15:43:23.884742 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:43:23.885182 24072 external_mini_cluster.cc:1428] Setting key b30d8a52d8e7eecaa9b045def18056aa
I20260808 15:43:23.891947 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:23.881665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:40711 (local address 127.23.130.62:45921)
0808 15:43:23.881897 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:23.881904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:23.883448 (+ 1544us) server_negotiation.cc:407] Connection header received
0808 15:43:23.885312 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:23.885317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:23.885406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:23.885538 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:23.887050 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.887939 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.889593 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.889984 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.890080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:23.891306 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:23.891311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:23.891314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:23.891331 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:23.891362 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:23.891374 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:23.891455 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:23.891641 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 15:43:23.891716 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:43:23.892896 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:23.882216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45921 (local address 127.23.130.3:40711)
0808 15:43:23.883289 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:23.883332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:23.885045 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:23.885767 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:23.885781 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:23.885923 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:23.886869 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:23.886889 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.888167 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.888174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:23.889465 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:23.889477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.890162 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.890167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:23.890362 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:23.891025 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:23.891027 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:23.891063 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:23.891137 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:23.891459 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:23.891467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:23.891472 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:43:23.891599 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 15:43:23.891881 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":361,"threads_started":1}
I20260808 15:43:23.894073 2537 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45921
I20260808 15:43:23.894358 2537 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:23.895077 2537 heartbeater.cc:511] Master 127.23.130.62:45921 requested a full tablet report, sending...
I20260808 15:43:23.896310 2039 ts_manager.cc:194] Registered new tserver with Master: 944494a0eed0425b93af9dabfe62496b (127.23.130.3:44697)
I20260808 15:43:23.897076 2039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.3:40711
I20260808 15:43:23.900530 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:43:23.906632 2246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:23.904934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.23.130.62:45921)
0808 15:43:23.905098 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:23.905102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:23.905590 (+ 488us) server_negotiation.cc:407] Connection header received
0808 15:43:23.905785 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:23.905791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:23.905869 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:23.905973 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:23.906438 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47782: BlockingRecv error: recv got EOF from 127.0.0.1:47782 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:43:23.907630 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2102
I20260808 15:43:23.915284 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:23.908532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.23.130.1:37931)
0808 15:43:23.909081 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:23.909094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:23.909119 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:43:23.909173 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:23.909179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:23.909385 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:23.909514 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:23.909834 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.910929 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.912153 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:23.912690 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:23.912801 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:23.913836 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:23.913844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:23.913854 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:23.914506 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:23.914552 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:23.914571 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:23.914658 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:23.914783 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 15:43:23.914931 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
I20260808 15:43:23.916469 2239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:23.997265 2244 heartbeater.cc:503] Master 127.23.130.62:45921 was elected leader, sending a full tablet report...
I20260808 15:43:23.998771 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2102
W20260808 15:43:24.018219 2023 connection.cc:535] server connection from 127.23.130.1:55295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:43:24.018559 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2248
I20260808 15:43:24.026501 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:24.019502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.23.130.2:35091)
0808 15:43:24.020066 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:24.020074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:24.020102 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:43:24.020180 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:24.020186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:24.020412 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:24.020541 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:24.020946 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:24.021892 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:24.023332 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:24.023905 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:24.024006 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:24.024996 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:24.025004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:24.025017 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:24.025607 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:24.025652 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:24.025671 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:24.025762 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:24.025905 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 15:43:24.026065 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
I20260808 15:43:24.027856 2382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:24.110888 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2248
I20260808 15:43:24.132026 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2391
I20260808 15:43:24.140676 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:24.133098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.23.130.3:44697)
0808 15:43:24.133721 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:24.133728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:24.133758 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:43:24.133817 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:24.133823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:24.134053 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:24.134228 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:24.134944 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:24.135987 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:24.137262 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:24.137848 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:24.138000 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:24.139205 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:24.139214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:24.139227 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:24.139807 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:24.139853 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:24.139871 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:24.140011 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:24.140190 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 15:43:24.140319 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":327,"threads_started":1}
I20260808 15:43:24.142031 2530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:24.236584 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2391
I20260808 15:43:24.254926 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2003
I20260808 15:43:24.256232 2073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:24.359588 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2003
2026-08-08T15:43:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2532 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T15:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:43:24Z Disabled control of system clock
I20260808 15:43:24.449985 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:46387
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:43:24.788444 2557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:24.788728 2557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:24.788815 2557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:24.796164 2557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:43:24.796265 2557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:24.796306 2557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:24.796365 2557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:43:24.796418 2557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:43:24.805465 2557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:46387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:24.806823 2557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:24.808589 2557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:24.820374 2563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:24.820485 2565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:24.820465 2567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:24.821491 2557 server_base.cc:1034] running on GCE node
I20260808 15:43:24.822250 2557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:24.823742 2557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:24.824977 2557 hybrid_clock.cc:648] HybridClock initialized: now 1786203804824929 us; error 34 us; skew 500 ppm
I20260808 15:43:24.825492 2557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:24.827955 2557 webserver.cc:460] Webserver started at http://127.23.130.62:36287/ using document root <none> and password file <none>
I20260808 15:43:24.828694 2557 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:24.828833 2557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:24.829202 2557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:24.834100 2557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "221c5ded42cf48d0bcf8ca2602c33930"
format_stamp: "Formatted at 2026-08-08 15:43:24 on dist-test-slave-gp6n"
server_key: "e4e3cb863636dfde62b787c0dca88ad2"
server_key_iv: "e278fdd7f16f60a7c960bddc94c5a209"
server_key_version: "encryptionkey@0"
I20260808 15:43:24.835094 2557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "221c5ded42cf48d0bcf8ca2602c33930"
format_stamp: "Formatted at 2026-08-08 15:43:24 on dist-test-slave-gp6n"
server_key: "e4e3cb863636dfde62b787c0dca88ad2"
server_key_iv: "e278fdd7f16f60a7c960bddc94c5a209"
server_key_version: "encryptionkey@0"
I20260808 15:43:24.840727 2557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:43:24.844496 2575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:24.845872 2557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:43:24.846064 2557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "221c5ded42cf48d0bcf8ca2602c33930"
format_stamp: "Formatted at 2026-08-08 15:43:24 on dist-test-slave-gp6n"
server_key: "e4e3cb863636dfde62b787c0dca88ad2"
server_key_iv: "e278fdd7f16f60a7c960bddc94c5a209"
server_key_version: "encryptionkey@0"
I20260808 15:43:24.846308 2557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:24.852491 2557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:24.853410 2557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:24.853745 2557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:24.874056 2557 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:46387
I20260808 15:43:24.874089 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:46387 every 8 connection(s)
I20260808 15:43:24.875625 2557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:43:24.879596 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:24.883837 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2557
I20260808 15:43:24.884092 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:43:24.884809 24072 external_mini_cluster.cc:1428] Setting key cec9e1ac1c1cf5f4489dadeaf682a0f8
I20260808 15:43:24.892102 2630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930: Bootstrap starting.
I20260808 15:43:24.894940 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:24.887014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.23.130.62:46387)
0808 15:43:24.888218 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:24.888233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:24.888287 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:43:24.890801 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:24.890829 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:24.891333 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:24.891807 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:24.892863 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:24.892872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:24.892889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:24.892907 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:24.892980 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:24.893007 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:24.893213 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:24.893342 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 15:43:24.893602 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":368,"threads_started":1}
I20260808 15:43:24.895987 2630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:24.897056 2630 log.cc:824] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:24.899895 2630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930: No bootstrap required, opened a new log
I20260808 15:43:24.907586 2630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } }
I20260808 15:43:24.907965 2630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:24.908077 2630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221c5ded42cf48d0bcf8ca2602c33930, State: Initialized, Role: FOLLOWER
I20260808 15:43:24.908712 2630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } }
I20260808 15:43:24.908962 2630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:43:24.909104 2630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:43:24.909291 2630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:24.911876 2630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } }
I20260808 15:43:24.912415 2630 leader_election.cc:302] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221c5ded42cf48d0bcf8ca2602c33930; no voters:
I20260808 15:43:24.913086 2630 leader_election.cc:288] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:43:24.913187 2636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:43:24.913946 2636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [term 1 LEADER]: Becoming Leader. State: Replica: 221c5ded42cf48d0bcf8ca2602c33930, State: Running, Role: LEADER
I20260808 15:43:24.914579 2636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } }
I20260808 15:43:24.915320 2630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:43:24.917662 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221c5ded42cf48d0bcf8ca2602c33930. Latest consensus state: current_term: 1 leader_uuid: "221c5ded42cf48d0bcf8ca2602c33930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } } }
I20260808 15:43:24.919736 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:24.921132 2648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:43:24.919489 2638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221c5ded42cf48d0bcf8ca2602c33930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c5ded42cf48d0bcf8ca2602c33930" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 46387 } } }
I20260808 15:43:24.921414 2638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:24.926010 2648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:43:24.934484 2648 catalog_manager.cc:1347] Generated new cluster ID: b17fe1be688849afb8b2b5ffee6ba6cc
I20260808 15:43:24.934592 2648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:43:24.966372 2648 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:43:24.966877 2648 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:43:24.975276 2648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221c5ded42cf48d0bcf8ca2602c33930: Generated new TSK 0
I20260808 15:43:24.976349 2648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:43:24.990218 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--builtin_ntp_servers=127.23.130.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:43:25.361399 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:25.361689 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:25.361857 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:25.370083 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:25.370189 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:25.370369 2658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:43:25.380184 2658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:25.381505 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:25.383342 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:25.396811 2667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:25.396785 2665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:25.397249 2664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:25.397589 2658 server_base.cc:1034] running on GCE node
I20260808 15:43:25.398233 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:25.399217 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:25.400460 2658 hybrid_clock.cc:648] HybridClock initialized: now 1786203805400409 us; error 45 us; skew 500 ppm
I20260808 15:43:25.400781 2658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:25.403704 2658 webserver.cc:460] Webserver started at http://127.23.130.1:46713/ using document root <none> and password file <none>
I20260808 15:43:25.404631 2658 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:25.404790 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:25.405139 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:25.408143 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "beb576a0ad8c4c62b3e1f07f3f15c93f"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "b0486be85355c3fa8267afb70aa0166e"
server_key_iv: "f7a7d533808817ae16a263cc7edd7927"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.408962 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "beb576a0ad8c4c62b3e1f07f3f15c93f"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "b0486be85355c3fa8267afb70aa0166e"
server_key_iv: "f7a7d533808817ae16a263cc7edd7927"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.414260 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:43:25.417892 2674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:25.419463 2658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:25.419673 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "beb576a0ad8c4c62b3e1f07f3f15c93f"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "b0486be85355c3fa8267afb70aa0166e"
server_key_iv: "f7a7d533808817ae16a263cc7edd7927"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.419960 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:25.426303 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:25.427304 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:25.427606 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:25.428563 2658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:25.430142 2658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:25.430263 2658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:25.430425 2658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:25.430536 2658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 15:43:25.465849 2658 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:37831
I20260808 15:43:25.465906 2802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:37831 every 8 connection(s)
I20260808 15:43:25.467329 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:43:25.471022 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2658
I20260808 15:43:25.471256 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:43:25.472157 24072 external_mini_cluster.cc:1428] Setting key 9a6241c2797fe9d0a84d859d208a3c44
I20260808 15:43:25.475185 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--builtin_ntp_servers=127.23.130.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:43:25.477084 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:25.470089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:57765 (local address 127.23.130.62:46387)
0808 15:43:25.470612 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:25.470619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:25.472272 (+ 1653us) server_negotiation.cc:407] Connection header received
0808 15:43:25.474354 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:25.474361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:25.474471 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:25.474588 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:25.476161 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:25.476170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:25.476178 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:25.476205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:25.476263 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:25.476282 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:25.476421 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:25.476635 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 15:43:25.476706 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":225,"threads_started":1}
I20260808 15:43:25.478045 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:25.470746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46387 (local address 127.23.130.1:57765)
0808 15:43:25.472048 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:25.472114 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:25.474030 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:25.474917 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:25.474931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:25.475044 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:25.475847 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:25.475850 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:25.475894 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:25.475988 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:25.476424 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:25.476433 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:25.476439 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:43:25.476669 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 15:43:25.476878 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":358,"threads_started":1}
I20260808 15:43:25.479691 2803 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46387
I20260808 15:43:25.480006 2803 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:25.480700 2803 heartbeater.cc:511] Master 127.23.130.62:46387 requested a full tablet report, sending...
I20260808 15:43:25.483063 2593 ts_manager.cc:194] Registered new tserver with Master: beb576a0ad8c4c62b3e1f07f3f15c93f (127.23.130.1:37831)
W20260808 15:43:25.811177 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:25.811439 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:25.811589 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:25.818833 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:25.818922 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:25.819119 2807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:43:25.828476 2807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:25.829756 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:25.831578 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:25.845382 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:25.845366 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:25.845942 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:25.846235 2807 server_base.cc:1034] running on GCE node
I20260808 15:43:25.846943 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:25.847925 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:25.849157 2807 hybrid_clock.cc:648] HybridClock initialized: now 1786203805849113 us; error 55 us; skew 500 ppm
I20260808 15:43:25.849530 2807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:25.851995 2807 webserver.cc:460] Webserver started at http://127.23.130.2:34227/ using document root <none> and password file <none>
I20260808 15:43:25.852679 2807 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:25.852813 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:25.853134 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:25.855994 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "293d38f862c0429bafb74bc367a29f95"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "040cf84f515687247704c1d9ba004d7a"
server_key_iv: "be884007d6cc55a04142aa8cf9d3fe2e"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.856782 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "293d38f862c0429bafb74bc367a29f95"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "040cf84f515687247704c1d9ba004d7a"
server_key_iv: "be884007d6cc55a04142aa8cf9d3fe2e"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.861956 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:43:25.865309 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:25.866715 2807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:25.866873 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "293d38f862c0429bafb74bc367a29f95"
format_stamp: "Formatted at 2026-08-08 15:43:25 on dist-test-slave-gp6n"
server_key: "040cf84f515687247704c1d9ba004d7a"
server_key_iv: "be884007d6cc55a04142aa8cf9d3fe2e"
server_key_version: "encryptionkey@0"
I20260808 15:43:25.867051 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:25.872375 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:25.873049 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:25.873348 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:25.874365 2807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:25.875794 2807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:25.875882 2807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:25.876034 2807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:25.876127 2807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:25.910447 2807 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:39563
I20260808 15:43:25.910503 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:39563 every 8 connection(s)
I20260808 15:43:25.911824 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:43:25.914805 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2807
I20260808 15:43:25.915022 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:43:25.915661 24072 external_mini_cluster.cc:1428] Setting key 2e26d2657b7cad0e5d2eebf3902a6750
I20260808 15:43:25.918861 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--builtin_ntp_servers=127.23.130.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:43:25.921687 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:25.914523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:57953 (local address 127.23.130.62:46387)
0808 15:43:25.914750 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:25.914758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:25.916585 (+ 1827us) server_negotiation.cc:407] Connection header received
0808 15:43:25.919048 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:25.919057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:25.919137 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:25.919235 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:25.920798 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:25.920802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:25.920806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:25.920821 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:25.920857 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:25.920903 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:25.921066 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:25.921325 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 15:43:25.921437 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:43:25.922935 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:25.915139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46387 (local address 127.23.130.2:57953)
0808 15:43:25.916321 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:25.916433 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:25.918681 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:25.919519 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:25.919533 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:25.919660 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:25.920388 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:25.920394 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:25.920460 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:25.920585 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:25.921057 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:25.921070 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:25.921081 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:43:25.921365 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 15:43:25.921652 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":326,"threads_started":1}
I20260808 15:43:25.924450 2950 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46387
I20260808 15:43:25.924772 2950 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:25.925503 2950 heartbeater.cc:511] Master 127.23.130.62:46387 requested a full tablet report, sending...
I20260808 15:43:25.927204 2593 ts_manager.cc:194] Registered new tserver with Master: 293d38f862c0429bafb74bc367a29f95 (127.23.130.2:39563)
W20260808 15:43:26.276384 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:26.276647 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:26.276800 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:26.283929 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:26.284029 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:26.284191 2954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:43:26.293146 2954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:26.294925 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:26.296868 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:26.310015 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:26.310081 2959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:26.310055 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:26.311132 2954 server_base.cc:1034] running on GCE node
I20260808 15:43:26.311687 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:26.312538 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:26.313735 2954 hybrid_clock.cc:648] HybridClock initialized: now 1786203806313698 us; error 24 us; skew 500 ppm
I20260808 15:43:26.314097 2954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:43:26.316500 2954 webserver.cc:460] Webserver started at http://127.23.130.3:45965/ using document root <none> and password file <none>
I20260808 15:43:26.317205 2954 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:26.317319 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:26.317659 2954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:26.320490 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "1a4388d6c7844c1ebf90f4ec1cb92993"
format_stamp: "Formatted at 2026-08-08 15:43:26 on dist-test-slave-gp6n"
server_key: "d64298e9d8bb6a4f29dc160d3f568063"
server_key_iv: "5aa3bb5c5f1a1910c02e3619fd030e21"
server_key_version: "encryptionkey@0"
I20260808 15:43:26.321376 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "1a4388d6c7844c1ebf90f4ec1cb92993"
format_stamp: "Formatted at 2026-08-08 15:43:26 on dist-test-slave-gp6n"
server_key: "d64298e9d8bb6a4f29dc160d3f568063"
server_key_iv: "5aa3bb5c5f1a1910c02e3619fd030e21"
server_key_version: "encryptionkey@0"
I20260808 15:43:26.326938 2954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:43:26.330364 2968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:26.331759 2954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:43:26.331943 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "1a4388d6c7844c1ebf90f4ec1cb92993"
format_stamp: "Formatted at 2026-08-08 15:43:26 on dist-test-slave-gp6n"
server_key: "d64298e9d8bb6a4f29dc160d3f568063"
server_key_iv: "5aa3bb5c5f1a1910c02e3619fd030e21"
server_key_version: "encryptionkey@0"
I20260808 15:43:26.332170 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:26.338109 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:26.339015 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:26.339289 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:26.340220 2954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:26.341660 2954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:26.341770 2954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:26.341908 2954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:26.342010 2954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:26.375613 2954 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:43701
I20260808 15:43:26.375671 3087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:43701 every 8 connection(s)
I20260808 15:43:26.376994 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:43:26.378229 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 2954
I20260808 15:43:26.378459 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:43:26.379076 24072 external_mini_cluster.cc:1428] Setting key fc68b2c3f291406503f63c27157caa49
I20260808 15:43:26.386029 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:26.379935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:43667 (local address 127.23.130.62:46387)
0808 15:43:26.380177 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:26.380184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:26.381798 (+ 1614us) server_negotiation.cc:407] Connection header received
0808 15:43:26.383733 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:26.383738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:26.383809 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:26.383925 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:26.385252 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:26.385258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:26.385263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:26.385284 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:26.385323 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:26.385335 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:26.385470 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:26.385666 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 15:43:26.385740 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:43:26.387071 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:26.380627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:46387 (local address 127.23.130.3:43667)
0808 15:43:26.381655 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:26.381690 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:26.383468 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:26.384210 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:26.384219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:26.384314 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:26.384941 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:43:26.384944 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:26.384995 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:43:26.385081 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:26.385434 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:26.385444 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:26.385452 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:43:26.385753 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 15:43:26.386024 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":358,"threads_started":1}
I20260808 15:43:26.388267 3088 heartbeater.cc:347] Connected to a master server at 127.23.130.62:46387
I20260808 15:43:26.388566 3088 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:26.389163 3088 heartbeater.cc:511] Master 127.23.130.62:46387 requested a full tablet report, sending...
I20260808 15:43:26.390715 2593 ts_manager.cc:194] Registered new tserver with Master: 1a4388d6c7844c1ebf90f4ec1cb92993 (127.23.130.3:43701)
I20260808 15:43:26.394562 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:43:26.400527 2805 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:43:26.399094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.23.130.62:46387)
0808 15:43:26.399251 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:26.399256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:26.399734 (+ 478us) server_negotiation.cc:407] Connection header received
0808 15:43:26.399867 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:26.399871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:26.399919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:26.399997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:26.400290 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:43:26.400966 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2658
I20260808 15:43:26.404637 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:26.402019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.23.130.1:37831)
0808 15:43:26.402589 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:26.402596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:26.402620 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:43:26.402674 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:26.402680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:26.402873 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:26.403004 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:26.403974 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:26.403982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:26.403994 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:26.404015 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:26.404061 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:26.404076 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:26.404184 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:26.404310 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 15:43:26.404391 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
I20260808 15:43:26.406189 2797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:26.487658 2803 heartbeater.cc:503] Master 127.23.130.62:46387 was elected leader, sending a full tablet report...
I20260808 15:43:26.489158 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2658
W20260808 15:43:26.508633 2578 connection.cc:535] server connection from 127.23.130.1:57765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:43:26.509019 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2807
I20260808 15:43:26.512784 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:26.509961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.23.130.2:39563)
0808 15:43:26.510521 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:26.510528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:26.510555 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:43:26.510617 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:26.510624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:26.510845 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:26.511014 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:26.512035 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:26.512041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:26.512052 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:26.512064 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:26.512101 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:26.512115 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:26.512258 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:26.512408 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 15:43:26.512514 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":241,"threads_started":1}
I20260808 15:43:26.514181 2944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:26.595412 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2807
I20260808 15:43:26.615684 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2954
I20260808 15:43:26.619988 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:26.616826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.23.130.3:43701)
0808 15:43:26.617447 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:26.617454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:26.617484 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:43:26.617548 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:26.617554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:26.617769 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:26.617944 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:26.619217 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:26.619223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:26.619234 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:43:26.619251 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:26.619298 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:43:26.619312 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:26.619498 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:26.619574 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 15:43:26.619676 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":300,"threads_started":1}
I20260808 15:43:26.621305 3083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:26.702837 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2954
I20260808 15:43:26.722281 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 2557
I20260808 15:43:26.723549 2625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:26.827782 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 2557
2026-08-08T15:43:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2466 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:43:26 dist-test-slave-gp6n krb5kdc[3109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:43:26 dist-test-slave-gp6n krb5kdc[3109](info): set up 2 sockets
Aug 08 15:43:26 dist-test-slave-gp6n krb5kdc[3109](info): commencing operation
krb5kdc: starting...
W20260808 15:43:28.022785 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:43:28 dist-test-slave-gp6n krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:43:28.306373 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37209
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:28.646329 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:28.646697 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:28.646876 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:28.653991 3125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:43:28.654160 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:28.654273 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:28.654381 3125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:43:28.654486 3125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:43:28.664130 3125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:28.665534 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:28.666939 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:28.679486 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:28.679540 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:28.679435 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:28.683810 3125 server_base.cc:1034] running on GCE node
I20260808 15:43:28.684619 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:28.686055 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:28.687292 3125 hybrid_clock.cc:648] HybridClock initialized: now 1786203808687248 us; error 55 us; skew 500 ppm
I20260808 15:43:28.687647 3125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:28 dist-test-slave-gp6n krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:28.695465 3125 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:43:28.697082 3125 webserver.cc:460] Webserver started at http://127.23.130.62:42549/ using document root <none> and password file <none>
I20260808 15:43:28.697806 3125 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:28.697959 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:28.698282 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:28.701470 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "14b518dc68f04cd0922bb8a402d90b73"
format_stamp: "Formatted at 2026-08-08 15:43:28 on dist-test-slave-gp6n"
server_key: "5e4af0fc3d966e0aa88f131920d84235"
server_key_iv: "6042a43fb26a484163451c71ca2280cf"
server_key_version: "encryptionkey@0"
I20260808 15:43:28.702253 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "14b518dc68f04cd0922bb8a402d90b73"
format_stamp: "Formatted at 2026-08-08 15:43:28 on dist-test-slave-gp6n"
server_key: "5e4af0fc3d966e0aa88f131920d84235"
server_key_iv: "6042a43fb26a484163451c71ca2280cf"
server_key_version: "encryptionkey@0"
I20260808 15:43:28.707770 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:43:28.711202 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:28.712522 3125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:43:28.712713 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "14b518dc68f04cd0922bb8a402d90b73"
format_stamp: "Formatted at 2026-08-08 15:43:28 on dist-test-slave-gp6n"
server_key: "5e4af0fc3d966e0aa88f131920d84235"
server_key_iv: "6042a43fb26a484163451c71ca2280cf"
server_key_version: "encryptionkey@0"
I20260808 15:43:28.712913 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:28.747678 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:28.751917 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:28.752230 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:28.771608 3125 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37209
I20260808 15:43:28.771644 3195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37209 every 8 connection(s)
I20260808 15:43:28.773116 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:43:28.774479 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 3125
I20260808 15:43:28.774710 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:43:28.775563 24072 external_mini_cluster.cc:1428] Setting key 7460dad617bc442082a539330af2681f
I20260808 15:43:28.778308 3196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:43:28 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:28.792145 3196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: Bootstrap starting.
I20260808 15:43:28.796502 3196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:28.797633 3196 log.cc:824] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:28.800700 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:28.777828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.23.130.62:37209)
0808 15:43:28.778709 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:28.778720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:28.778761 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:43:28.780399 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:28.780419 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:28.780834 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:28.781176 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:28.783142 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:28.784501 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:28.786320 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:28.787069 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:28.787204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:28.792039 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:28.792049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:28.792069 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:28.792124 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:28.796018 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:28.796701 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:28.796708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:28.796714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:28.796887 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:28.797285 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:28.797290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:28.797293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:28.797882 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:28.798237 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:28.798744 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 15:43:28.799245 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":383,"threads_started":1}
I20260808 15:43:28.801364 3196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: No bootstrap required, opened a new log
I20260808 15:43:28.809391 3196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } }
I20260808 15:43:28.809760 3196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:28.809842 3196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b518dc68f04cd0922bb8a402d90b73, State: Initialized, Role: FOLLOWER
I20260808 15:43:28.810442 3196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } }
I20260808 15:43:28.810678 3196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:43:28.810783 3196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:43:28.810911 3196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:28.813246 3196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } }
I20260808 15:43:28.813720 3196 leader_election.cc:302] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14b518dc68f04cd0922bb8a402d90b73; no voters:
I20260808 15:43:28.814283 3196 leader_election.cc:288] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:43:28.814459 3201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:43:28.815182 3201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [term 1 LEADER]: Becoming Leader. State: Replica: 14b518dc68f04cd0922bb8a402d90b73, State: Running, Role: LEADER
I20260808 15:43:28.815729 3201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } }
I20260808 15:43:28.816541 3196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:43:28.818909 3203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14b518dc68f04cd0922bb8a402d90b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } } }
I20260808 15:43:28.819402 3203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:28.820322 3204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b518dc68f04cd0922bb8a402d90b73. Latest consensus state: current_term: 1 leader_uuid: "14b518dc68f04cd0922bb8a402d90b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b518dc68f04cd0922bb8a402d90b73" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37209 } } }
I20260808 15:43:28.820780 3204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73 [sys.catalog]: This master's current role is: LEADER
I20260808 15:43:28.821813 3213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:43:28.826115 3213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:43:28.834645 3213 catalog_manager.cc:1347] Generated new cluster ID: f1778bd1c5a743198965ec9b55f7cd08
I20260808 15:43:28.834772 3213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:43:28.848994 3213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:43:28.850267 3213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:43:28.861383 3213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: Generated new TSK 0
I20260808 15:43:28.862505 3213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:43:28.934778 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:29.280974 3226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:29.281245 3226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:29.281407 3226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:29.288667 3226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:29.288774 3226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:29.288996 3226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:43:29.298777 3226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:29.300279 3226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:29.301949 3226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:29.315645 3232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:29.315636 3233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:29.315636 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:29.316558 3226 server_base.cc:1034] running on GCE node
I20260808 15:43:29.317219 3226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:29.318275 3226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:29.319479 3226 hybrid_clock.cc:648] HybridClock initialized: now 1786203809319457 us; error 100 us; skew 500 ppm
I20260808 15:43:29.319788 3226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:29 dist-test-slave-gp6n krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:29.323395 3226 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:43:29.324816 3226 webserver.cc:460] Webserver started at http://127.23.130.1:35829/ using document root <none> and password file <none>
I20260808 15:43:29.325519 3226 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:29.325655 3226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:29.325968 3226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:29.328998 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "4d3231188d6a4574b2bb310508132cb2"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "4aee6f9b7dab967d5c21b07aebddfd11"
server_key_iv: "f91e0998c3df732b17a081575fd277c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.329805 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3231188d6a4574b2bb310508132cb2"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "4aee6f9b7dab967d5c21b07aebddfd11"
server_key_iv: "f91e0998c3df732b17a081575fd277c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.335074 3226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:43:29.338624 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:29.339972 3226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:43:29.340163 3226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "4d3231188d6a4574b2bb310508132cb2"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "4aee6f9b7dab967d5c21b07aebddfd11"
server_key_iv: "f91e0998c3df732b17a081575fd277c2"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.340396 3226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:29.355304 3226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:29.359565 3226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:29.359898 3226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:29.360924 3226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:29.362488 3226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:29.362592 3226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:29.362739 3226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:29.362815 3226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:29.396166 3226 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:36625
I20260808 15:43:29.396214 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:36625 every 8 connection(s)
I20260808 15:43:29.397496 3226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:43:29.404552 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 3226
I20260808 15:43:29.404767 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:43:29.405616 24072 external_mini_cluster.cc:1428] Setting key 60c445b15781bc57760b9a50c1f7d73b
Aug 08 15:43:29 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:29.420779 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:29.400145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:43253 (local address 127.23.130.62:37209)
0808 15:43:29.400666 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:29.400673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:29.401629 (+ 956us) server_negotiation.cc:407] Connection header received
0808 15:43:29.403446 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:29.403451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:29.403518 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:29.403617 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:29.405184 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.406328 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.408405 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.409111 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.409289 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:29.412768 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:29.412776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:29.412784 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:29.412829 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:29.416404 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:29.417767 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:29.417775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:29.417781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:29.417985 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:29.418577 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:29.418584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:29.418590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:29.418957 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:29.419122 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:29.420168 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0808 15:43:29.420402 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":243,"threads_started":1}
I20260808 15:43:29.421752 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:29.400719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37209 (local address 127.23.130.1:43253)
0808 15:43:29.401447 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:29.401495 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:29.403184 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:29.403826 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:29.403834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:29.404315 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:29.405004 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:29.405026 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.406523 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.406528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:29.408239 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:29.408251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.409354 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.409360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:29.409569 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:29.410264 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:29.410287 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:29.412548 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:29.416855 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:29.416863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:29.416934 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:29.417637 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:29.418128 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:29.418135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:29.418141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:29.418437 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:29.419117 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:29.419126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:29.419637 (+ 511us) client_negotiation.cc:769] Sending connection context
0808 15:43:29.420017 (+ 380us) client_negotiation.cc:240] Negotiation successful
0808 15:43:29.420554 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
I20260808 15:43:29.423329 3372 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37209
I20260808 15:43:29.423653 3372 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:29.424394 3372 heartbeater.cc:511] Master 127.23.130.62:37209 requested a full tablet report, sending...
I20260808 15:43:29.427033 3158 ts_manager.cc:194] Registered new tserver with Master: 4d3231188d6a4574b2bb310508132cb2 (127.23.130.1:36625)
I20260808 15:43:29.429248 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:43253
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:43:29.470243 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:29.804443 3379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:29.804716 3379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:29.804811 3379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:29.812589 3379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:29.812700 3379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:29.812810 3379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:43:29.822839 3379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:29.824059 3379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:29.825311 3379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:29.839007 3385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:29.839000 3384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:29.839269 3387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:29.840106 3379 server_base.cc:1034] running on GCE node
I20260808 15:43:29.840778 3379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:29.841801 3379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:29.843045 3379 hybrid_clock.cc:648] HybridClock initialized: now 1786203809842980 us; error 46 us; skew 500 ppm
I20260808 15:43:29.843374 3379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:29 dist-test-slave-gp6n krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:29.846781 3379 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:43:29.848232 3379 webserver.cc:460] Webserver started at http://127.23.130.2:42557/ using document root <none> and password file <none>
I20260808 15:43:29.848943 3379 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:29.849105 3379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:29.849478 3379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:29.852532 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "06b212b03d2c4ef6b00bdee27036db8b"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "dd4df2bf000652fac0153159459c1ad8"
server_key_iv: "cbc609562543eac7fb93a20baa906767"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.853310 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "06b212b03d2c4ef6b00bdee27036db8b"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "dd4df2bf000652fac0153159459c1ad8"
server_key_iv: "cbc609562543eac7fb93a20baa906767"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.858649 3379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:43:29.862057 3395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:29.863390 3379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:43:29.863565 3379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "06b212b03d2c4ef6b00bdee27036db8b"
format_stamp: "Formatted at 2026-08-08 15:43:29 on dist-test-slave-gp6n"
server_key: "dd4df2bf000652fac0153159459c1ad8"
server_key_iv: "cbc609562543eac7fb93a20baa906767"
server_key_version: "encryptionkey@0"
I20260808 15:43:29.863775 3379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:29.923225 3379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:29.927409 3379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:29.927723 3379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:29.928705 3379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:29.930109 3379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:29.930214 3379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:29.930363 3379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:29.930465 3379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 15:43:29.964437 3379 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:43135
I20260808 15:43:29.964530 3517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:43135 every 8 connection(s)
I20260808 15:43:29.965760 3379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:43:29.968467 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 3379
I20260808 15:43:29.968670 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:43:29.969331 24072 external_mini_cluster.cc:1428] Setting key f767d8952a2c78d0ea3f1b736fb630f2
Aug 08 15:43:29 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:29.989907 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:29.968741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:58961 (local address 127.23.130.62:37209)
0808 15:43:29.969288 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:29.969297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:29.970310 (+ 1013us) server_negotiation.cc:407] Connection header received
0808 15:43:29.972692 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:29.972696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:29.972760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:29.972853 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:29.975132 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.975884 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.978352 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.978872 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.978972 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:29.983326 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:29.983331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:29.983334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:29.983368 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:29.986575 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:29.987378 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:29.987384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:29.987387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:29.987540 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:29.987991 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:29.987996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:29.987999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:29.988285 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:29.988448 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:29.989244 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 15:43:29.989505 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260808 15:43:29.991040 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:29.969293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37209 (local address 127.23.130.2:58961)
0808 15:43:29.970136 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:29.970182 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:29.972369 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:29.973142 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:29.973156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:29.974006 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:29.974929 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:29.974960 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.976044 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.976052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:29.978206 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:29.978225 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:29.979104 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:29.979113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:29.979462 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:29.980209 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:29.980245 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:29.983075 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:29.986740 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:29.986747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:29.986761 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:29.987239 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:29.987661 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:29.987668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:29.987676 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:29.987866 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:29.988421 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:29.988429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:29.988840 (+ 411us) client_negotiation.cc:769] Sending connection context
0808 15:43:29.989134 (+ 294us) client_negotiation.cc:240] Negotiation successful
0808 15:43:29.989727 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":345,"threads_started":1}
I20260808 15:43:29.992643 3518 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37209
I20260808 15:43:29.993027 3518 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:29.993988 3518 heartbeater.cc:511] Master 127.23.130.62:37209 requested a full tablet report, sending...
I20260808 15:43:29.995795 3158 ts_manager.cc:194] Registered new tserver with Master: 06b212b03d2c4ef6b00bdee27036db8b (127.23.130.2:43135)
I20260808 15:43:29.996683 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:58961
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:43:30.038394 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:43:30.397074 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:43:30.397397 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:43:30.397575 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:43:30.405081 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:43:30.405231 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:43:30.405439 3524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:43:30.415316 3524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:43:30.416769 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:43:30.418268 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:43:30.431533 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:30.432318 3372 heartbeater.cc:503] Master 127.23.130.62:37209 was elected leader, sending a full tablet report...
W20260808 15:43:30.432364 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:43:30.433451 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:43:30.433836 3524 server_base.cc:1034] running on GCE node
I20260808 15:43:30.434598 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:43:30.435596 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:43:30.436792 3524 hybrid_clock.cc:648] HybridClock initialized: now 1786203810436733 us; error 64 us; skew 500 ppm
I20260808 15:43:30.437109 3524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:43:30 dist-test-slave-gp6n krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:43:30.440335 3524 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:43:30.441745 3524 webserver.cc:460] Webserver started at http://127.23.130.3:34817/ using document root <none> and password file <none>
I20260808 15:43:30.442528 3524 fs_manager.cc:359] Metadata directory not provided
I20260808 15:43:30.442672 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:43:30.442977 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:43:30.446024 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "4de10afd550149458064669aab84ca50"
format_stamp: "Formatted at 2026-08-08 15:43:30 on dist-test-slave-gp6n"
server_key: "2643ef2de62b8c4d4de052d4caedca33"
server_key_iv: "f4a0cc10d16053f322cc6370f3eabf3f"
server_key_version: "encryptionkey@0"
I20260808 15:43:30.446820 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "4de10afd550149458064669aab84ca50"
format_stamp: "Formatted at 2026-08-08 15:43:30 on dist-test-slave-gp6n"
server_key: "2643ef2de62b8c4d4de052d4caedca33"
server_key_iv: "f4a0cc10d16053f322cc6370f3eabf3f"
server_key_version: "encryptionkey@0"
I20260808 15:43:30.452140 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:43:30.455639 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:30.457041 3524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:43:30.457240 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "4de10afd550149458064669aab84ca50"
format_stamp: "Formatted at 2026-08-08 15:43:30 on dist-test-slave-gp6n"
server_key: "2643ef2de62b8c4d4de052d4caedca33"
server_key_iv: "f4a0cc10d16053f322cc6370f3eabf3f"
server_key_version: "encryptionkey@0"
I20260808 15:43:30.457448 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:43:30.501614 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:43:30.505877 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:43:30.506191 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:43:30.507434 3524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:43:30.508833 3524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:43:30.508945 3524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:30.509097 3524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:43:30.509173 3524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:43:30.544209 3524 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:42855
I20260808 15:43:30.544310 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:42855 every 8 connection(s)
I20260808 15:43:30.545576 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:43:30.551788 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 3524
I20260808 15:43:30.552013 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:43:30.552522 24072 external_mini_cluster.cc:1428] Setting key 0c69c507cc01a66767ca78fee0c7e019
Aug 08 15:43:30 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:43:30.566515 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.548363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59413 (local address 127.23.130.62:37209)
0808 15:43:30.548872 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.548879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.549705 (+ 826us) server_negotiation.cc:407] Connection header received
0808 15:43:30.551605 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.551612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.551713 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.551867 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:30.553698 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.554769 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.556711 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.557111 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.557208 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.560636 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:30.560641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:30.560644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:30.560676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:30.563650 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.564376 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.564380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.564382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.564505 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.564895 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.564898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.564900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.565152 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:30.565268 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.565894 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.566084 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":220,"threads_started":1}
I20260808 15:43:30.567643 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.548750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37209 (local address 127.23.130.3:59413)
0808 15:43:30.549535 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.549580 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.551302 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.552163 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.552172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.552752 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:43:30.553513 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.553543 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.554922 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.554928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.556574 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.556591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.557293 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.557298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.557507 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.558212 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:43:30.558235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:43:30.560445 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:43:30.563775 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:30.563781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:30.563792 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:30.564232 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:30.564600 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:43:30.564603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:43:30.564605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:43:30.564781 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:43:30.565260 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:43:30.565265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:43:30.565576 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.565801 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.566187 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":304,"threads_started":1}
I20260808 15:43:30.568969 3664 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37209
I20260808 15:43:30.569288 3664 heartbeater.cc:464] Registering TS with master...
I20260808 15:43:30.570007 3664 heartbeater.cc:511] Master 127.23.130.62:37209 requested a full tablet report, sending...
I20260808 15:43:30.571470 3158 ts_manager.cc:194] Registered new tserver with Master: 4de10afd550149458064669aab84ca50 (127.23.130.3:42855)
I20260808 15:43:30.572270 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:59413
I20260808 15:43:30.580554 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:43:30.597500 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.585323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.23.130.62:37209)
0808 15:43:30.585521 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.585526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.586030 (+ 504us) server_negotiation.cc:407] Connection header received
0808 15:43:30.586157 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.586161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.586225 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.586317 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:30.587779 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.588544 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.589813 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.590277 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.590383 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.592648 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:30.592652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:30.592656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:30.592685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:30.595111 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.595710 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.595714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.595716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.595834 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.596231 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.596234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.596236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.596581 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:30.596763 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.597094 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.597255 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:43:30.601692 3158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:43:30.609313 3158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:43:30.634478 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.622575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:43135 (local address 127.0.0.1:42522)
0808 15:43:30.623328 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.623373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.623526 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.624890 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.624901 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.624943 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:43:30.625443 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.625458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.627578 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.627586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.630559 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.630574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.633316 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.633326 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.633621 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.633690 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.633821 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.634018 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":392,"threads_started":1}
I20260808 15:43:30.635566 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.622638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.23.130.2:43135)
0808 15:43:30.623930 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.623941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.623967 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:43:30.624063 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.624070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.624327 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.624544 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:43:30.625701 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.627403 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.630745 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.632691 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.632859 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.633124 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.634902 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.635077 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":1025,"threads_started":1}
I20260808 15:43:30.643148 3447 tablet_service.cc:1467] Processing CreateTablet for tablet b1cf315bff5846a0932fa66dcad5db62 (DEFAULT_TABLE table=test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:43:30.644464 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.622607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:42855 (local address 127.0.0.1:50914)
0808 15:43:30.634923 (+ 12316us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.634948 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.635103 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.636298 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.636306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.636341 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:43:30.636789 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.636803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.639479 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.639483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.641651 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.641668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.643507 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.643513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.643686 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.643719 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.643823 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.643983 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12150,"thread_start_us":11986,"threads_started":1}
I20260808 15:43:30.644538 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.623156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.23.130.3:42855)
0808 15:43:30.634229 (+ 11073us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.634242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.635262 (+ 1020us) server_negotiation.cc:407] Connection header received
0808 15:43:30.635386 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.635397 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.635707 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.635943 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:43:30.637027 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.639155 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.641811 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.643290 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.643434 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.643677 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.643944 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.644175 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10852,"thread_start_us":11074,"threads_started":1}
I20260808 15:43:30.644899 3447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1cf315bff5846a0932fa66dcad5db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:30.646171 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.624354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36625 (local address 127.0.0.1:42502)
0808 15:43:30.636088 (+ 11734us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.636110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.636228 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.639908 (+ 3680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.639913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.639943 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:43:30.640155 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.640171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.641574 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.641579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.643447 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.643461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.645558 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.645564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.645750 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.645781 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.645883 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.645997 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11610,"thread_start_us":14890,"threads_started":1}
I20260808 15:43:30.646870 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.624799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.23.130.1:36625)
0808 15:43:30.639020 (+ 14221us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.639031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.639063 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:43:30.639160 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.639176 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.639416 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.639613 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:43:30.640345 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.641420 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.643634 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.645112 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.645245 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.645473 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.646523 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.646644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14035,"thread_start_us":13844,"threads_started":1}
I20260808 15:43:30.648705 3590 tablet_service.cc:1467] Processing CreateTablet for tablet b1cf315bff5846a0932fa66dcad5db62 (DEFAULT_TABLE table=test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:43:30.649705 3297 tablet_service.cc:1467] Processing CreateTablet for tablet b1cf315bff5846a0932fa66dcad5db62 (DEFAULT_TABLE table=test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:43:30.650235 3590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1cf315bff5846a0932fa66dcad5db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:30.650918 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1cf315bff5846a0932fa66dcad5db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:43:30.660028 3683 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Bootstrap starting.
I20260808 15:43:30.662736 3684 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Bootstrap starting.
I20260808 15:43:30.664218 3683 tablet_bootstrap.cc:654] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:30.664611 3685 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Bootstrap starting.
I20260808 15:43:30.665546 3683 log.cc:824] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:30.666291 3684 tablet_bootstrap.cc:654] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:30.667515 3684 log.cc:824] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:30.668133 3685 tablet_bootstrap.cc:654] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Neither blocks nor log segments found. Creating new log.
I20260808 15:43:30.668777 3683 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: No bootstrap required, opened a new log
I20260808 15:43:30.669061 3683 ts_tablet_manager.cc:1397] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260808 15:43:30.669317 3685 log.cc:824] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Log is configured to *not* fsync() on all Append() calls
I20260808 15:43:30.670506 3684 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: No bootstrap required, opened a new log
I20260808 15:43:30.670771 3684 ts_tablet_manager.cc:1397] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260808 15:43:30.672254 3685 tablet_bootstrap.cc:492] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: No bootstrap required, opened a new log
I20260808 15:43:30.672499 3685 ts_tablet_manager.cc:1397] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:43:30.676653 3683 raft_consensus.cc:348] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.677030 3683 raft_consensus.cc:374] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:30.677196 3683 raft_consensus.cc:729] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b212b03d2c4ef6b00bdee27036db8b, State: Initialized, Role: FOLLOWER
I20260808 15:43:30.677901 3683 consensus_queue.cc:260] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.678993 3518 heartbeater.cc:503] Master 127.23.130.62:37209 was elected leader, sending a full tablet report...
I20260808 15:43:30.679661 3683 ts_tablet_manager.cc:1428] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260808 15:43:30.681370 3684 raft_consensus.cc:348] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.681785 3684 raft_consensus.cc:374] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:30.681891 3684 raft_consensus.cc:729] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3231188d6a4574b2bb310508132cb2, State: Initialized, Role: FOLLOWER
I20260808 15:43:30.682569 3684 consensus_queue.cc:260] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.683009 3685 raft_consensus.cc:348] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.683523 3685 raft_consensus.cc:374] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:43:30.683665 3685 raft_consensus.cc:729] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de10afd550149458064669aab84ca50, State: Initialized, Role: FOLLOWER
I20260808 15:43:30.684356 3684 ts_tablet_manager.cc:1428] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260808 15:43:30.684404 3685 consensus_queue.cc:260] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.685808 3664 heartbeater.cc:503] Master 127.23.130.62:37209 was elected leader, sending a full tablet report...
I20260808 15:43:30.686501 3685 ts_tablet_manager.cc:1428] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260808 15:43:30.717845 3519 tablet.cc:2369] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:43:30.797363 3665 tablet.cc:2369] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:43:30.831947 3692 raft_consensus.cc:482] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:43:30.832274 3692 raft_consensus.cc:504] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.834066 3692 leader_election.cc:288] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06b212b03d2c4ef6b00bdee27036db8b (127.23.130.2:43135), 4d3231188d6a4574b2bb310508132cb2 (127.23.130.1:36625)
I20260808 15:43:30.840801 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.834613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:46405 (local address 127.23.130.2:43135)
0808 15:43:30.834803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.834808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.834848 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:43:30.835012 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.835019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.835107 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.835217 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:43:30.835835 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.836877 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.838784 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.839879 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.839991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.840050 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.840428 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.840528 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:43:30.840837 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.834496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:43135 (local address 127.23.130.3:46405)
0808 15:43:30.834698 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.834721 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.834861 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.835429 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.835435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.835471 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:43:30.835644 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.835652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.837002 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.837007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.838634 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.838646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.840025 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.840031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.840223 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.840263 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.840358 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.840504 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 15:43:30.841439 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.835231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:47605 (local address 127.23.130.1:36625)
0808 15:43:30.835433 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.835439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.835740 (+ 301us) server_negotiation.cc:407] Connection header received
0808 15:43:30.835918 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.835921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.835978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.836051 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:43:30.836596 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.837551 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.839507 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.840529 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.840677 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.840745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.841132 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.841241 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:43:30.842092 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.834921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36625 (local address 127.23.130.3:47605)
0808 15:43:30.835637 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 15:43:30.835651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:43:30.835778 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:43:30.836094 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:43:30.836099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:43:30.836124 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:43:30.836450 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.836461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.837696 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.837700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.839344 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:43:30.839358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.840729 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.840735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:43:30.840929 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.840963 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:43:30.841626 (+ 663us) client_negotiation.cc:240] Negotiation successful
0808 15:43:30.841756 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":416,"threads_started":1}
I20260808 15:43:30.841966 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1cf315bff5846a0932fa66dcad5db62" candidate_uuid: "4de10afd550149458064669aab84ca50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b212b03d2c4ef6b00bdee27036db8b" is_pre_election: true
I20260808 15:43:30.842623 3471 raft_consensus.cc:2454] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4de10afd550149458064669aab84ca50 in term 0.
I20260808 15:43:30.843310 3543 leader_election.cc:302] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b212b03d2c4ef6b00bdee27036db8b, 4de10afd550149458064669aab84ca50; no voters:
I20260808 15:43:30.843223 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1cf315bff5846a0932fa66dcad5db62" candidate_uuid: "4de10afd550149458064669aab84ca50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3231188d6a4574b2bb310508132cb2" is_pre_election: true
I20260808 15:43:30.843744 3692 raft_consensus.cc:2781] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:43:30.843926 3692 raft_consensus.cc:482] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:43:30.843860 3319 raft_consensus.cc:2454] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4de10afd550149458064669aab84ca50 in term 0.
I20260808 15:43:30.844064 3692 raft_consensus.cc:3037] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:30.846535 3692 raft_consensus.cc:504] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.847185 3692 leader_election.cc:288] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [CANDIDATE]: Term 1 election: Requested vote from peers 06b212b03d2c4ef6b00bdee27036db8b (127.23.130.2:43135), 4d3231188d6a4574b2bb310508132cb2 (127.23.130.1:36625)
I20260808 15:43:30.847611 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1cf315bff5846a0932fa66dcad5db62" candidate_uuid: "4de10afd550149458064669aab84ca50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b212b03d2c4ef6b00bdee27036db8b"
I20260808 15:43:30.847651 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1cf315bff5846a0932fa66dcad5db62" candidate_uuid: "4de10afd550149458064669aab84ca50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3231188d6a4574b2bb310508132cb2"
I20260808 15:43:30.847823 3471 raft_consensus.cc:3037] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:30.847836 3319 raft_consensus.cc:3037] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:43:30.849989 3319 raft_consensus.cc:2454] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4de10afd550149458064669aab84ca50 in term 1.
I20260808 15:43:30.850225 3471 raft_consensus.cc:2454] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4de10afd550149458064669aab84ca50 in term 1.
I20260808 15:43:30.850425 3543 leader_election.cc:302] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3231188d6a4574b2bb310508132cb2, 4de10afd550149458064669aab84ca50; no voters:
I20260808 15:43:30.850706 3692 raft_consensus.cc:2781] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:43:30.851174 3692 raft_consensus.cc:686] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 1 LEADER]: Becoming Leader. State: Replica: 4de10afd550149458064669aab84ca50, State: Running, Role: LEADER
I20260808 15:43:30.851630 3692 consensus_queue.cc:237] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } }
I20260808 15:43:30.856010 3157 catalog_manager.cc:5495] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4de10afd550149458064669aab84ca50 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "4de10afd550149458064669aab84ca50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4de10afd550149458064669aab84ca50" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 } health_report { overall_health: UNKNOWN } } }
I20260808 15:43:30.872256 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.866558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.23.130.3:42855)
0808 15:43:30.866911 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.866916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.866935 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:43:30.867010 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.867013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.867086 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.867277 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:43:30.867619 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.868594 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.870011 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.870530 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.870635 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.871151 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:43:30.871694 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:43:30.871811 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.871942 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.872008 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260808 15:43:30.880429 3319 raft_consensus.cc:1261] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 1 FOLLOWER]: Refusing update from remote peer 4de10afd550149458064669aab84ca50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:43:30.880437 3471 raft_consensus.cc:1261] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 1 FOLLOWER]: Refusing update from remote peer 4de10afd550149458064669aab84ca50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:43:30.881455 3694 consensus_queue.cc:1035] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b212b03d2c4ef6b00bdee27036db8b" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:43:30.881901 3692 consensus_queue.cc:1035] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3231188d6a4574b2bb310508132cb2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:43:30.910171 3157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:43:30.910526 3157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:43:30.914459 3157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14b518dc68f04cd0922bb8a402d90b73: Sending DeleteTablet for 3 replicas of tablet b1cf315bff5846a0932fa66dcad5db62
I20260808 15:43:30.915758 3297 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cf315bff5846a0932fa66dcad5db62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:43:30 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:42502
I20260808 15:43:30.915865 3590 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cf315bff5846a0932fa66dcad5db62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:43:30 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:50914
I20260808 15:43:30.916625 3447 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cf315bff5846a0932fa66dcad5db62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:43:30 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:42522
I20260808 15:43:30.917532 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 3226
I20260808 15:43:30.926800 3708 tablet_replica.cc:330] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: stopping tablet replica
I20260808 15:43:30.927382 3708 raft_consensus.cc:2229] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:43:30.928005 3708 raft_consensus.cc:2258] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:43:30.930290 3710 tablet_replica.cc:330] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: stopping tablet replica
I20260808 15:43:30.930775 3710 raft_consensus.cc:2229] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:43:30.931339 3710 raft_consensus.cc:2258] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:43:30.932202 3709 tablet_replica.cc:330] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: stopping tablet replica
I20260808 15:43:30.932618 3709 raft_consensus.cc:2229] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:43:30.933368 3709 raft_consensus.cc:2258] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:43:30.934314 3710 ts_tablet_manager.cc:1905] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:43:30.935019 3708 ts_tablet_manager.cc:1905] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:43:30.938210 3709 ts_tablet_manager.cc:1905] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:43:30 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:43:30.942461 3708 ts_tablet_manager.cc:1918] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:43:30.942742 3708 log.cc:1196] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/b1cf315bff5846a0932fa66dcad5db62
I20260808 15:43:30.943446 3708 ts_tablet_manager.cc:1939] T b1cf315bff5846a0932fa66dcad5db62 P 4d3231188d6a4574b2bb310508132cb2: Deleting consensus metadata
I20260808 15:43:30.943490 3710 ts_tablet_manager.cc:1918] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:43:30.943756 3710 log.cc:1196] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/b1cf315bff5846a0932fa66dcad5db62
I20260808 15:43:30.944480 3710 ts_tablet_manager.cc:1939] T b1cf315bff5846a0932fa66dcad5db62 P 06b212b03d2c4ef6b00bdee27036db8b: Deleting consensus metadata
I20260808 15:43:30.945658 3144 catalog_manager.cc:4841] TS 4d3231188d6a4574b2bb310508132cb2 (127.23.130.1:36625): tablet b1cf315bff5846a0932fa66dcad5db62 (table test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]) successfully deleted
I20260808 15:43:30.946455 3144 catalog_manager.cc:4841] TS 06b212b03d2c4ef6b00bdee27036db8b (127.23.130.2:43135): tablet b1cf315bff5846a0932fa66dcad5db62 (table test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]) successfully deleted
I20260808 15:43:30.948483 3709 ts_tablet_manager.cc:1918] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:43:30.948730 3709 log.cc:1196] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/b1cf315bff5846a0932fa66dcad5db62
I20260808 15:43:30.949213 3709 ts_tablet_manager.cc:1939] T b1cf315bff5846a0932fa66dcad5db62 P 4de10afd550149458064669aab84ca50: Deleting consensus metadata
I20260808 15:43:30.950682 3143 catalog_manager.cc:4841] TS 4de10afd550149458064669aab84ca50 (127.23.130.3:42855): tablet b1cf315bff5846a0932fa66dcad5db62 (table test-table [id=ad6a66cfacda48fc82ebe348d0a9df37]) successfully deleted
I20260808 15:43:30.955161 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:30.919197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.23.130.1:36625)
0808 15:43:30.919439 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:30.919448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:30.929744 (+ 10296us) server_negotiation.cc:407] Connection header received
0808 15:43:30.930191 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:30.930198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:30.930296 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:30.930394 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:30.932440 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.933597 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.935495 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:30.936175 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:30.936289 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:30.947328 (+ 11039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:30.947343 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:30.947361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:30.947412 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:30.951769 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.952485 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.952496 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.952500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.952677 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:30.953046 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:30.953052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:30.953056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:30.953584 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:30.953939 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:30.954116 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 15:43:30.954363 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:43:30.955964 3366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:31.058879 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 3226
I20260808 15:43:31.081007 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 3379
Aug 08 15:43:31 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:43:31.099045 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:31.081946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.23.130.2:43135)
0808 15:43:31.082153 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:31.082161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:31.082189 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:43:31.082328 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:31.082334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:31.082466 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:31.082583 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:31.083860 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:31.085229 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:31.086575 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:31.087246 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:31.087399 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:31.091684 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:31.091709 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:31.091732 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:31.091797 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:31.096023 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:31.096693 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:31.096702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:31.096706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:31.096871 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:31.097274 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:31.097279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:31.097283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:31.097772 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:31.098073 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:31.098393 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 15:43:31.098647 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:43:31.099757 3513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:31.192425 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 3379
I20260808 15:43:31.219592 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 3524
Aug 08 15:43:31 dist-test-slave-gp6n krb5kdc[3109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:43:31.238878 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:43:31.220534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.23.130.3:42855)
0808 15:43:31.220748 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:43:31.220752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:43:31.220771 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:43:31.221004 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:43:31.221008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:43:31.221079 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:43:31.221170 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:43:31.223071 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:31.223881 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:31.225742 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:43:31.226200 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:43:31.226311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:43:31.232035 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:43:31.232050 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:43:31.232066 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:43:31.232105 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:43:31.235716 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:31.236482 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:31.236496 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:31.236499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:31.236654 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:43:31.237078 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:43:31.237085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:43:31.237089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:43:31.237494 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:43:31.237924 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0808 15:43:31.238200 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:43:31.238485 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:43:31.239723 3659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:31.363087 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 3524
I20260808 15:43:31.385236 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 3125
I20260808 15:43:31.386377 3191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:43:31.498833 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 3125
2026-08-08T15:43:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4672 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:43:31 dist-test-slave-gp6n krb5kdc[3718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:43:31 dist-test-slave-gp6n krb5kdc[3718](info): set up 2 sockets
Aug 08 15:43:31 dist-test-slave-gp6n krb5kdc[3718](info): commencing operation
krb5kdc: starting...
W20260808 15:43:32.685127 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:43:32 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:43:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangeradmin_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangeradmin_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangerlookup_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangerlookup_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
I20260808 15:43:33.093158 24072 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2oCF2B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/postgres -l logfile start
2026-08-08 15:43:35.742 UTC [3753] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:43:35.742 UTC [3753] LOG: listening on IPv4 address "127.23.130.20", port 60155
2026-08-08 15:43:35.746 UTC [3753] LOG: listening on Unix socket "/tmp/.s.PGSQL.60155"
2026-08-08 15:43:35.769 UTC [3756] LOG: database system was shut down at 2026-08-08 15:43:34 UTC
2026-08-08 15:43:35.773 UTC [3753] LOG: database system is ready to accept connections
I20260808 15:43:36.810283 24072 mini_postgres.cc:96] Postgres bound to 60155
2026-08-08 15:43:36.835 UTC [3764] FATAL: database "slave" does not exist
127.23.130.20:60155 - accepting connections
I20260808 15:43:36.836714 24072 mini_ranger.cc:161] Starting Ranger...
I20260808 15:43:36.868695 24072 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:43:37.359341 24072 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:43:37.359535 24072 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin
2026-08-08 15:43:37,690 [I] DB FLAVOR :POSTGRES
2026-08-08 15:43:37,691 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:43:37,691 [I] Checking connection..
2026-08-08 15:43:37,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:43:37,955 [I] Checking connection passed.
2026-08-08 15:43:37,955 [I] --------- Verifying version history table ---------
2026-08-08 15:43:37,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:43:38,231 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:43:38,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:43:38,518 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:43:38,518 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:43:38,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:43:38.797 UTC [3832] WARNING: there is no transaction in progress
2026-08-08 15:43:38,803 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:43:38,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:43:39,075 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:43:39,075 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:43:39,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:43:39,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:43:39,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-08 15:43:39,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:43:40,093 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:43:40,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:43:40,355 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:43:40,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:43:40,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:43:40,859 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:43:40,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:43:41.802 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.818 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.835 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.850 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.864 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.881 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.933 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.940 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.954 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.964 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.975 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.989 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:41.999 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:42.005 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:42.012 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:42.018 UTC [3948] WARNING: there is no transaction in progress
2026-08-08 15:43:42,670 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:43:42,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:43:42,905 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:43:42,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:43:43,171 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:43:43,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:43:43,435 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:43:43,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:43:43,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:43:43,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-08 15:43:44,189 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:43:44,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:43:44,422 [I] DB_PATCHES have already been applied
I20260808 15:43:44.428371 24072 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2oCF2B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:43:44.428488 24072 mini_ranger.cc:193] Using host: 127.23.130.20
I20260808 15:43:44.449784 24072 mini_ranger.cc:227] Ranger admin URL: http://127.23.130.20:50101
Aug 08, 2026 3:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:43:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:43:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:43:45.701566 24072 mini_ranger.cc:160] Time spent starting Ranger: real 8.865s user 0.001s sys 0.062s
Aug 08, 2026 3:44:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:44:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 3:44:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:44:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:44:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:44:07.041086 24072 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangerkms_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/rangerkms_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.130.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.130.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:44:07 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203847, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:44:07.253238 24072 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:44:07.280767 24072 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:44:08.183032 24072 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:44:08.183173 24072 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms
I20260808 15:44:08.183343 24072 mini_ranger_kms.cc:227] Using postgres at 127.23.130.20:60155
2026-08-08 15:44:08,716 [I] DB FLAVOR :POSTGRES
2026-08-08 15:44:08,717 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:44:08,717 [I] Checking connection
2026-08-08 15:44:08,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:44:08,999 [I] connection success
2026-08-08 15:44:09,000 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:44:09,000 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:44:09,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:44:09,300 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:44:09,301 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:44:09,301 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:44:09,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:60155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:44:09,630 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:44:10.129071 24072 mini_ranger_kms.cc:310] Created kms service
I20260808 15:44:10.335577 24072 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:44:10.380549 24072 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:44:10 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203847, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:10.788244 24072 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:44:10.788650 24072 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms:/tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:44:10.788725 24072 mini_ranger_kms.cc:240] Using host: 127.23.130.20
I20260808 15:44:10.817097 24072 mini_ranger_kms.cc:276] Ranger KMS PID: 4142
I20260808 15:44:10.817318 24072 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.130.20:34181
Aug 08, 2026 3:44:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:44:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:44:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:44:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:44:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:44:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 15:44:13 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786203853, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:44:13.305907 24072 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.053s user 0.129s sys 0.416s
I20260808 15:44:13.306178 24072 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:44:13.306383 24072 mini_ranger_kms.cc:401] 127.23.130.20:34181/kms/v1/keys
Aug 08 15:44:13 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203853, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
Aug 08 15:44:14 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203853, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
Aug 08 15:44:16 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:44:16.595580 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39527
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:39527
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:44:16.960803 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:44:16.961127 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:44:16.961360 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:44:16.969424 4199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:44:16.969583 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:44:16.969671 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:44:16.969777 4199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:44:16.969867 4199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:44:16.980594 4199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.130.20:34181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:39527
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:44:16.981915 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:44:16.983314 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:44:16.996603 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:16.996685 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:16.996656 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:44:16.997246 4199 server_base.cc:1034] running on GCE node
I20260808 15:44:16.998066 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:44:16.999552 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:44:17.000783 4199 hybrid_clock.cc:648] HybridClock initialized: now 1786203857000739 us; error 41 us; skew 500 ppm
I20260808 15:44:17.001142 4199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:44:17 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203857, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:44:17.004515 4199 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:44:17.005995 4199 webserver.cc:460] Webserver started at http://127.23.130.62:33101/ using document root <none> and password file <none>
I20260808 15:44:17.006749 4199 fs_manager.cc:359] Metadata directory not provided
I20260808 15:44:17.006917 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:44:17.007273 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:44:17 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203857, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:17.072575 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "24011d2e21c84b1a8bd023b5d7667312"
format_stamp: "Formatted at 2026-08-08 15:44:17 on dist-test-slave-gp6n"
server_key: "78480591a6900d63457bafcec758c9ea"
server_key_iv: "c7aa792f0bf7ab0264e92974a5005d50"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:17.073755 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "24011d2e21c84b1a8bd023b5d7667312"
format_stamp: "Formatted at 2026-08-08 15:44:17 on dist-test-slave-gp6n"
server_key: "78480591a6900d63457bafcec758c9ea"
server_key_iv: "c7aa792f0bf7ab0264e92974a5005d50"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:17.081192 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260808 15:44:17.101649 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:17.103585 4199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:44:17.103768 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "24011d2e21c84b1a8bd023b5d7667312"
format_stamp: "Formatted at 2026-08-08 15:44:17 on dist-test-slave-gp6n"
server_key: "78480591a6900d63457bafcec758c9ea"
server_key_iv: "c7aa792f0bf7ab0264e92974a5005d50"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:17.104024 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:44:17.124223 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:44:17.128690 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:44:17.129026 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:44:17.149729 4199 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:39527
I20260808 15:44:17.149763 4271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:39527 every 8 connection(s)
I20260808 15:44:17.151183 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:44:17.155048 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:44:17.160969 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 4199
I20260808 15:44:17.161190 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:44:17.161847 24072 external_mini_cluster.cc:1428] Setting key 52622fbb8cba27496f5185e4ed72e3c0
I20260808 15:44:17.165416 4272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: Bootstrap starting.
I20260808 15:44:17.169046 4272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: Neither blocks nor log segments found. Creating new log.
I20260808 15:44:17.170131 4272 log.cc:824] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: Log is configured to *not* fsync() on all Append() calls
I20260808 15:44:17.173035 4272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: No bootstrap required, opened a new log
Aug 08 15:44:17 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:44:17.181720 4272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } }
I20260808 15:44:17.182124 4272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:44:17.182250 4272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24011d2e21c84b1a8bd023b5d7667312, State: Initialized, Role: FOLLOWER
I20260808 15:44:17.182933 4272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } }
I20260808 15:44:17.183189 4272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:44:17.183349 4272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:44:17.183557 4272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:44:17.185439 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:17.163855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.23.130.62:39527)
0808 15:44:17.164665 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:17.164677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:17.164717 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:44:17.166606 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:17.166628 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:17.166997 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:17.167332 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:17.168958 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:17.170460 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:17.172063 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:17.172709 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:17.172835 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:17.177618 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:17.177627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:17.177644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:17.177734 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:17.180751 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:17.181416 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:17.181434 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:17.181441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:17.181587 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:17.181979 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:17.181985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:17.181988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:17.182782 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:17.183108 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:17.183597 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 15:44:17.184063 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":402,"threads_started":1}
I20260808 15:44:17.186620 4272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } }
I20260808 15:44:17.187152 4272 leader_election.cc:302] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24011d2e21c84b1a8bd023b5d7667312; no voters:
I20260808 15:44:17.187819 4272 leader_election.cc:288] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:44:17.187966 4278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:44:17.189288 4278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [term 1 LEADER]: Becoming Leader. State: Replica: 24011d2e21c84b1a8bd023b5d7667312, State: Running, Role: LEADER
I20260808 15:44:17.189806 4278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } }
I20260808 15:44:17.190809 4272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:44:17.192755 4279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24011d2e21c84b1a8bd023b5d7667312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } } }
I20260808 15:44:17.193236 4279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [sys.catalog]: This master's current role is: LEADER
I20260808 15:44:17.194612 4280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24011d2e21c84b1a8bd023b5d7667312. Latest consensus state: current_term: 1 leader_uuid: "24011d2e21c84b1a8bd023b5d7667312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24011d2e21c84b1a8bd023b5d7667312" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39527 } } }
I20260808 15:44:17.194965 4280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312 [sys.catalog]: This master's current role is: LEADER
I20260808 15:44:17.196223 4287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:44:17.202389 4287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:44:17.203251 4272 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:44:17.223053 4287 catalog_manager.cc:1347] Generated new cluster ID: 039ec8605ac9426489b8b45cb4a6e3dd
I20260808 15:44:17.223210 4287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:44:17.242883 4287 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:44:17.244889 4287 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:44:17.256735 4287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: Generated new TSK 0
I20260808 15:44:17.257879 4287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:44:18.019070 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:44:18 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:44:18.444649 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:44:18.445047 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:44:18.445209 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:44:18.457114 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:44:18.457263 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:44:18.457461 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:44:18.473273 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.130.20:34181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:44:18.474998 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:44:18.476756 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:44:18.494673 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:18.495169 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:44:18.500522 4320 server_base.cc:1034] running on GCE node
W20260808 15:44:18.501283 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:44:18.502317 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:44:18.506723 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:44:18.508011 4320 hybrid_clock.cc:648] HybridClock initialized: now 1786203858507924 us; error 85 us; skew 500 ppm
I20260808 15:44:18.508388 4320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:44:18 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:44:18.512373 4320 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:44:18.515888 4320 webserver.cc:460] Webserver started at http://127.23.130.1:40997/ using document root <none> and password file <none>
I20260808 15:44:18.516629 4320 fs_manager.cc:359] Metadata directory not provided
I20260808 15:44:18.516776 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:44:18.517123 4320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:44:18 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:18.534974 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "6762eb4a3028462d8c33cbab9f315251"
format_stamp: "Formatted at 2026-08-08 15:44:18 on dist-test-slave-gp6n"
server_key: "f623c8aa291b5959893b9d10102e5096"
server_key_iv: "600fed164e6aeee030c750125194d7e0"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:18.536831 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "6762eb4a3028462d8c33cbab9f315251"
format_stamp: "Formatted at 2026-08-08 15:44:18 on dist-test-slave-gp6n"
server_key: "f623c8aa291b5959893b9d10102e5096"
server_key_iv: "600fed164e6aeee030c750125194d7e0"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:18.552155 4320 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.013s
I20260808 15:44:18.570796 4341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:18.572446 4320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:44:18.572649 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "6762eb4a3028462d8c33cbab9f315251"
format_stamp: "Formatted at 2026-08-08 15:44:18 on dist-test-slave-gp6n"
server_key: "f623c8aa291b5959893b9d10102e5096"
server_key_iv: "600fed164e6aeee030c750125194d7e0"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:18.572930 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:44:18.617949 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:44:18.623667 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:44:18.624117 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:44:18.625442 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:44:18.627400 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:44:18.627498 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:18.627617 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:44:18.627688 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:44:18 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:19.037676 4320 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:46697
I20260808 15:44:19.037740 4465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:46697 every 8 connection(s)
I20260808 15:44:19.039384 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:44:19.042793 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 4320
I20260808 15:44:19.043011 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:44:19.043927 24072 external_mini_cluster.cc:1428] Setting key dc09e28003317373a311b73a3a047abc
Aug 08 15:44:19 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
I20260808 15:44:19.090395 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:19.043195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39527 (local address 127.23.130.1:49097)
0808 15:44:19.044222 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:19.044301 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:19.046794 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:19.063372 (+ 16578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:19.063390 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:19.064216 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:44:19.065295 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:19.065329 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.067216 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.067223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:19.074871 (+ 7648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:19.074893 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.076053 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.076060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:19.076390 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:19.077224 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:44:19.077257 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:44:19.080125 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:44:19.085089 (+ 4964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:19.085099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:19.085119 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:19.085787 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:19.086285 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:19.086290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:19.086293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:19.086647 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:19.087461 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:44:19.087469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:44:19.087968 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 15:44:19.088287 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 15:44:19.088879 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":995,"threads_started":1}
I20260808 15:44:19.091468 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:19.044063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49097 (local address 127.23.130.62:39527)
0808 15:44:19.062578 (+ 18515us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:19.062585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:19.062611 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:44:19.062689 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:19.062695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:19.062791 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:19.062934 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:19.065611 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.066979 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.075105 (+ 8126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.075769 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.076869 (+ 1100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:19.080445 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:19.080452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:19.080457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:19.080512 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:19.084888 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:19.085961 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:19.085966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:19.085969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:19.086130 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:19.086799 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:19.086805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:19.086809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:19.087208 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:19.087367 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:19.090779 (+ 3412us) server_negotiation.cc:299] Negotiation successful
0808 15:44:19.091028 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18366,"thread_start_us":18993,"threads_started":1}
I20260808 15:44:19.093657 4466 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39527
I20260808 15:44:19.094026 4466 heartbeater.cc:464] Registering TS with master...
I20260808 15:44:19.095183 4466 heartbeater.cc:511] Master 127.23.130.62:39527 requested a full tablet report, sending...
I20260808 15:44:19.098061 4234 ts_manager.cc:194] Registered new tserver with Master: 6762eb4a3028462d8c33cbab9f315251 (127.23.130.1:46697)
I20260808 15:44:19.100414 4234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:49097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:44:19.126228 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:44:19 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
W20260808 15:44:19.601970 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:44:19.602293 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:44:19.602528 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:44:19.610179 4474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:44:19.610354 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:44:19.610560 4474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:44:19.621819 4474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.130.20:34181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:44:19.623235 4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:44:19.624644 4474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:44:19.638146 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:19.638706 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:19.639371 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:44:19.639899 4474 server_base.cc:1034] running on GCE node
I20260808 15:44:19.640513 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:44:19.641479 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:44:19.642711 4474 hybrid_clock.cc:648] HybridClock initialized: now 1786203859642652 us; error 53 us; skew 500 ppm
I20260808 15:44:19.643033 4474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:44:19 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:44:19.646510 4474 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:44:19.648089 4474 webserver.cc:460] Webserver started at http://127.23.130.2:36727/ using document root <none> and password file <none>
I20260808 15:44:19.648764 4474 fs_manager.cc:359] Metadata directory not provided
I20260808 15:44:19.648895 4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:44:19.649231 4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:44:19 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:19.666615 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "34e808b3b00042fcaf68279ed3c1133c"
format_stamp: "Formatted at 2026-08-08 15:44:19 on dist-test-slave-gp6n"
server_key: "418f5dd559e99627961df7bd902c5689"
server_key_iv: "5d8199a93a1b6f756c1796aafd1a6445"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:19.667799 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "34e808b3b00042fcaf68279ed3c1133c"
format_stamp: "Formatted at 2026-08-08 15:44:19 on dist-test-slave-gp6n"
server_key: "418f5dd559e99627961df7bd902c5689"
server_key_iv: "5d8199a93a1b6f756c1796aafd1a6445"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:19.675369 4474 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260808 15:44:19.696725 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:19.698539 4474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:44:19.698712 4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "34e808b3b00042fcaf68279ed3c1133c"
format_stamp: "Formatted at 2026-08-08 15:44:19 on dist-test-slave-gp6n"
server_key: "418f5dd559e99627961df7bd902c5689"
server_key_iv: "5d8199a93a1b6f756c1796aafd1a6445"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:19.698992 4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:44:19.718572 4474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:44:19.723261 4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:44:19.723583 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:44:19.724673 4474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:44:19.726357 4474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:44:19.726466 4474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 15:44:19.726588 4474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:44:19.726760 4474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:19.767671 4474 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:46295
I20260808 15:44:19.767691 4619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:46295 every 8 connection(s)
I20260808 15:44:19.769338 4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:44:19.770016 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 4474
I20260808 15:44:19.770211 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:44:19.770819 24072 external_mini_cluster.cc:1428] Setting key 6ba577ff73c3bc0dbc37dd97ba067ca3
Aug 08 15:44:19 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:44:19.809538 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:19.773010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:34703 (local address 127.23.130.62:39527)
0808 15:44:19.790867 (+ 17857us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:19.790874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:19.792085 (+ 1211us) server_negotiation.cc:407] Connection header received
0808 15:44:19.794184 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:19.794191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:19.794285 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:19.794459 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:19.796009 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.797058 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.798785 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.799365 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.799508 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:19.803181 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:19.803187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:19.803190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:19.803222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:19.806146 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:19.806952 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:19.806957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:19.806961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:19.807094 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:19.807533 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:19.807538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:19.807546 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:19.807824 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:19.807986 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:19.808773 (+ 787us) server_negotiation.cc:299] Negotiation successful
0808 15:44:19.809071 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17721,"thread_start_us":17612,"threads_started":1}
I20260808 15:44:19.810738 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:19.772965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39527 (local address 127.23.130.2:34703)
0808 15:44:19.791449 (+ 18484us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:19.791513 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:19.793806 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:19.794717 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:19.794726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:19.795217 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:44:19.795817 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:19.795835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.797228 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.797233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:19.798640 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:19.798652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:19.799604 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:19.799614 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:19.799879 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:19.800718 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:44:19.800751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:44:19.802923 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:44:19.806279 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:19.806287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:19.806305 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:19.806820 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:19.807203 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:19.807209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:19.807212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:19.807411 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:19.807977 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:44:19.807985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:44:19.808323 (+ 338us) client_negotiation.cc:769] Sending connection context
0808 15:44:19.808606 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 15:44:19.809169 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18170,"thread_start_us":17957,"threads_started":1}
I20260808 15:44:19.812397 4620 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39527
I20260808 15:44:19.812775 4620 heartbeater.cc:464] Registering TS with master...
I20260808 15:44:19.813725 4620 heartbeater.cc:511] Master 127.23.130.62:39527 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
I20260808 15:44:19.815550 4234 ts_manager.cc:194] Registered new tserver with Master: 34e808b3b00042fcaf68279ed3c1133c (127.23.130.2:46295)
I20260808 15:44:19.816574 4234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:34703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:44:19.854816 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:44:20.104595 4466 heartbeater.cc:503] Master 127.23.130.62:39527 was elected leader, sending a full tablet report...
W20260808 15:44:20.225627 4626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:44:20.225932 4626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:44:20.226029 4626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:44:20.233959 4626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:44:20.234083 4626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:44:20.234200 4626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:44:20.244653 4626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.130.20:34181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:44:20.246003 4626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:44:20.247342 4626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:44:20.260509 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:20.260682 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:44:20.260727 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:44:20.261710 4626 server_base.cc:1034] running on GCE node
I20260808 15:44:20.262326 4626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:44:20.263268 4626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:44:20.264575 4626 hybrid_clock.cc:648] HybridClock initialized: now 1786203860264520 us; error 40 us; skew 500 ppm
I20260808 15:44:20.264891 4626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:44:20 dist-test-slave-gp6n krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:44:20.268205 4626 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:44:20.269663 4626 webserver.cc:460] Webserver started at http://127.23.130.3:33495/ using document root <none> and password file <none>
I20260808 15:44:20.270329 4626 fs_manager.cc:359] Metadata directory not provided
I20260808 15:44:20.270498 4626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:44:20.270825 4626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:44:20 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:44:20.285293 4626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "a5b3d6f31cee4192976521c0ddf3b4cc"
format_stamp: "Formatted at 2026-08-08 15:44:20 on dist-test-slave-gp6n"
server_key: "dbdf0a5026fc749ec979cd9e32d56ca5"
server_key_iv: "b921ed879ca06c7f9e962c46034996bf"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:20.286242 4626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b3d6f31cee4192976521c0ddf3b4cc"
format_stamp: "Formatted at 2026-08-08 15:44:20 on dist-test-slave-gp6n"
server_key: "dbdf0a5026fc749ec979cd9e32d56ca5"
server_key_iv: "b921ed879ca06c7f9e962c46034996bf"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:20.292400 4626 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 15:44:20.307555 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:20.309322 4626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:44:20.309506 4626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "a5b3d6f31cee4192976521c0ddf3b4cc"
format_stamp: "Formatted at 2026-08-08 15:44:20 on dist-test-slave-gp6n"
server_key: "dbdf0a5026fc749ec979cd9e32d56ca5"
server_key_iv: "b921ed879ca06c7f9e962c46034996bf"
server_key_version: "kuduclusterkey@0"
I20260808 15:44:20.309785 4626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:44:20.334213 4626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:44:20.338938 4626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:44:20.339324 4626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:44:20.340412 4626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:44:20.341934 4626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:44:20.342027 4626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:20.342129 4626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:44:20.342180 4626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:44:20.382249 4626 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:41695
I20260808 15:44:20.382283 4766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:41695 every 8 connection(s)
I20260808 15:44:20.383703 4626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:44:20.384944 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 4626
I20260808 15:44:20.385147 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:44:20.385711 24072 external_mini_cluster.cc:1428] Setting key f1f5207a0cd65eb4e353e7b418ff468f
Aug 08 15:44:20 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203860, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:44:20.405970 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.386804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:56575 (local address 127.23.130.62:39527)
0808 15:44:20.387397 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.387404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.388695 (+ 1291us) server_negotiation.cc:407] Connection header received
0808 15:44:20.390978 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.390982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.391061 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.391157 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:20.392642 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.393361 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.395045 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.395469 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.395565 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.399058 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:20.399064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:20.399067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:20.399099 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:20.403055 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.403898 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.403919 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.403925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.404076 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.404450 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.404454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.404456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.404723 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:20.404915 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.405540 (+ 625us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.405718 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":313,"threads_started":1}
I20260808 15:44:20.407012 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.387537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:39527 (local address 127.23.130.3:56575)
0808 15:44:20.388516 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.388572 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.390682 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.391410 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.391418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.391917 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:44:20.392495 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.392513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.393519 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.393523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.394902 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.394914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.395658 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.395663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.395864 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.396498 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:44:20.396520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:44:20.398836 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:44:20.403243 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:20.403252 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:20.403269 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:20.403777 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:20.404174 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:44:20.404177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:44:20.404179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:44:20.404355 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:44:20.404889 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:44:20.404895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:44:20.405221 (+ 326us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.405439 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.405816 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":417,"threads_started":1}
I20260808 15:44:20.408351 4767 heartbeater.cc:347] Connected to a master server at 127.23.130.62:39527
I20260808 15:44:20.408672 4767 heartbeater.cc:464] Registering TS with master...
I20260808 15:44:20.409502 4767 heartbeater.cc:511] Master 127.23.130.62:39527 requested a full tablet report, sending...
I20260808 15:44:20.410972 4234 ts_manager.cc:194] Registered new tserver with Master: a5b3d6f31cee4192976521c0ddf3b4cc (127.23.130.3:41695)
I20260808 15:44:20.411796 4234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:56575
I20260808 15:44:20.414148 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:44:20.431716 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.419101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.23.130.62:39527)
0808 15:44:20.419257 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.419262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.419874 (+ 612us) server_negotiation.cc:407] Connection header received
0808 15:44:20.420010 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.420014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.420092 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.420188 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:20.421598 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.422380 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.423717 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.424122 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.424210 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.426659 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:20.426664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:20.426667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:20.426697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:20.429265 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.429959 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.429963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.429965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.430081 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.430569 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.430572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.430574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.430837 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:20.431025 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.431319 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.431489 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:44:20.435854 4234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:44:20.443420 4234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:44:20.466931 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.456379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.23.130.3:41695)
0808 15:44:20.457987 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.457998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.458033 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:44:20.458133 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.458140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.458382 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.458635 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:44:20.459523 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.461147 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.464422 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.465709 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.465840 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.466069 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.466372 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.466575 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1242,"threads_started":1}
I20260808 15:44:20.466982 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.456111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:41695 (local address 127.0.0.1:48520)
0808 15:44:20.456959 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.457028 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.457246 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.458650 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.458657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.458694 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:44:20.459155 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.459174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.461275 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.461282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.464237 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.464252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.465903 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.465909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.466125 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.466174 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.466320 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.466541 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":62,"thread_start_us":342,"threads_started":1}
I20260808 15:44:20.468432 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.457295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:46295 (local address 127.0.0.1:38586)
0808 15:44:20.458650 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.458671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.458820 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.459548 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.459554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.459581 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:44:20.460867 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.460882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.462737 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.462742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.464982 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.464996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.467059 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.467065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.467255 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.467289 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.467380 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.467506 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":1060,"threads_started":1}
I20260808 15:44:20.468515 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.459261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46697 (local address 127.0.0.1:44562)
0808 15:44:20.460615 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.460628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.460768 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.461358 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.461365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.461398 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:44:20.461868 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.461879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.463576 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.463581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.465483 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.465499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.467598 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.467604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.467776 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.467808 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.467902 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.468031 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":4,"thread_start_us":1095,"threads_started":1}
I20260808 15:44:20.468546 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.458991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.23.130.1:46697)
0808 15:44:20.460109 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.460121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.460729 (+ 608us) server_negotiation.cc:407] Connection header received
0808 15:44:20.460939 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.460946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.461171 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.461373 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:44:20.462100 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.463393 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.465664 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.467065 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.467221 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.467424 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.467973 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.468161 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":782,"threads_started":1}
I20260808 15:44:20.470086 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.457663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.23.130.2:46295)
0808 15:44:20.458215 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.458222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.458964 (+ 742us) server_negotiation.cc:407] Connection header received
0808 15:44:20.459080 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.459089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.459328 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.460980 (+ 1652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:44:20.461201 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.462596 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.465144 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.466574 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.466708 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.466897 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.469622 (+ 2725us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.469796 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":280,"threads_started":1}
I20260808 15:44:20.471915 4393 tablet_service.cc:1467] Processing CreateTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 (DEFAULT_TABLE table=test-table [id=a5643efcfb56441f854879bc66de6e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:44:20.471837 4693 tablet_service.cc:1467] Processing CreateTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 (DEFAULT_TABLE table=test-table [id=a5643efcfb56441f854879bc66de6e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:44:20.473299 4393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1f0744884d44ecbd6b0e7a267a0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:44:20.473322 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1f0744884d44ecbd6b0e7a267a0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:44:20.473484 4550 tablet_service.cc:1467] Processing CreateTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 (DEFAULT_TABLE table=test-table [id=a5643efcfb56441f854879bc66de6e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:44:20.474802 4550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1f0744884d44ecbd6b0e7a267a0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:44:20.485098 4790 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Bootstrap starting.
I20260808 15:44:20.489717 4790 tablet_bootstrap.cc:654] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Neither blocks nor log segments found. Creating new log.
I20260808 15:44:20.489818 4792 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Bootstrap starting.
I20260808 15:44:20.491300 4790 log.cc:824] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Log is configured to *not* fsync() on all Append() calls
I20260808 15:44:20.491618 4791 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Bootstrap starting.
I20260808 15:44:20.493650 4792 tablet_bootstrap.cc:654] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Neither blocks nor log segments found. Creating new log.
I20260808 15:44:20.495038 4791 tablet_bootstrap.cc:654] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Neither blocks nor log segments found. Creating new log.
I20260808 15:44:20.495031 4792 log.cc:824] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:44:20.495381 4790 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: No bootstrap required, opened a new log
I20260808 15:44:20.495793 4790 ts_tablet_manager.cc:1397] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260808 15:44:20.496320 4791 log.cc:824] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:44:20.498533 4792 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: No bootstrap required, opened a new log
I20260808 15:44:20.498795 4792 ts_tablet_manager.cc:1397] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:44:20.499102 4791 tablet_bootstrap.cc:492] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: No bootstrap required, opened a new log
I20260808 15:44:20.499351 4791 ts_tablet_manager.cc:1397] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260808 15:44:20.507035 4791 raft_consensus.cc:348] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.507588 4791 raft_consensus.cc:374] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:44:20.507728 4791 raft_consensus.cc:729] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b3d6f31cee4192976521c0ddf3b4cc, State: Initialized, Role: FOLLOWER
I20260808 15:44:20.508358 4791 consensus_queue.cc:260] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.508769 4790 raft_consensus.cc:348] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.509326 4790 raft_consensus.cc:374] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:44:20.509485 4790 raft_consensus.cc:729] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762eb4a3028462d8c33cbab9f315251, State: Initialized, Role: FOLLOWER
I20260808 15:44:20.509505 4767 heartbeater.cc:503] Master 127.23.130.62:39527 was elected leader, sending a full tablet report...
I20260808 15:44:20.510281 4791 ts_tablet_manager.cc:1428] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260808 15:44:20.510336 4790 consensus_queue.cc:260] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.510655 4792 raft_consensus.cc:348] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.511179 4792 raft_consensus.cc:374] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:44:20.511340 4792 raft_consensus.cc:729] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e808b3b00042fcaf68279ed3c1133c, State: Initialized, Role: FOLLOWER
I20260808 15:44:20.512094 4792 consensus_queue.cc:260] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.513489 4620 heartbeater.cc:503] Master 127.23.130.62:39527 was elected leader, sending a full tablet report...
I20260808 15:44:20.514673 4792 ts_tablet_manager.cc:1428] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260808 15:44:20.518846 4790 ts_tablet_manager.cc:1428] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
W20260808 15:44:20.533797 4621 tablet.cc:2369] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:44:20.551548 4467 tablet.cc:2369] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:44:20.635464 4768 tablet.cc:2369] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:44:20.728849 4799 raft_consensus.cc:482] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:44:20.729205 4799 raft_consensus.cc:504] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.731045 4799 leader_election.cc:288] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b3d6f31cee4192976521c0ddf3b4cc (127.23.130.3:41695), 6762eb4a3028462d8c33cbab9f315251 (127.23.130.1:46697)
I20260808 15:44:20.738344 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.731441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:58151 (local address 127.23.130.3:41695)
0808 15:44:20.731691 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.731700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.732353 (+ 653us) server_negotiation.cc:407] Connection header received
0808 15:44:20.732610 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.732617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.732700 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.732776 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:44:20.733663 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.734503 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.736437 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.737293 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.737420 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.737515 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.737934 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.738067 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:44:20.738355 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.732464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:48287 (local address 127.23.130.1:46697)
0808 15:44:20.732691 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.732697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.732946 (+ 249us) server_negotiation.cc:407] Connection header received
0808 15:44:20.733059 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.733063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.733145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.733247 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:44:20.733833 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.734659 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.736594 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.737387 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.737564 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.737639 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.737992 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.738123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:44:20.738487 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.731471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:41695 (local address 127.23.130.2:58151)
0808 15:44:20.732204 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.732229 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.732405 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.733007 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.733020 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.733049 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:44:20.733501 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.733516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.734646 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.734651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.736265 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.736277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.737528 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.737535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.737738 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.737794 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.737868 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.738002 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":72,"thread_start_us":384,"threads_started":1}
I20260808 15:44:20.738541 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.732237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:46697 (local address 127.23.130.2:48287)
0808 15:44:20.732838 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 15:44:20.732852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:44:20.732960 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:44:20.733265 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:44:20.733268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:44:20.733285 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:44:20.733673 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.733683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.734835 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.734839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.736434 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:44:20.736449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.737627 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.737631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:44:20.737808 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.737830 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:44:20.737946 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:44:20.738066 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":43,"thread_start_us":330,"threads_started":1}
I20260808 15:44:20.739722 4715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1f0744884d44ecbd6b0e7a267a0cb7" candidate_uuid: "34e808b3b00042fcaf68279ed3c1133c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" is_pre_election: true
I20260808 15:44:20.739749 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1f0744884d44ecbd6b0e7a267a0cb7" candidate_uuid: "34e808b3b00042fcaf68279ed3c1133c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762eb4a3028462d8c33cbab9f315251" is_pre_election: true
I20260808 15:44:20.740293 4415 raft_consensus.cc:2454] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e808b3b00042fcaf68279ed3c1133c in term 0.
I20260808 15:44:20.740293 4715 raft_consensus.cc:2454] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e808b3b00042fcaf68279ed3c1133c in term 0.
I20260808 15:44:20.740935 4504 leader_election.cc:302] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e808b3b00042fcaf68279ed3c1133c, 6762eb4a3028462d8c33cbab9f315251; no voters:
I20260808 15:44:20.741328 4799 raft_consensus.cc:2781] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:44:20.741494 4799 raft_consensus.cc:482] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:44:20.741631 4799 raft_consensus.cc:3037] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:44:20.744074 4799 raft_consensus.cc:504] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.744693 4799 leader_election.cc:288] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [CANDIDATE]: Term 1 election: Requested vote from peers a5b3d6f31cee4192976521c0ddf3b4cc (127.23.130.3:41695), 6762eb4a3028462d8c33cbab9f315251 (127.23.130.1:46697)
I20260808 15:44:20.745157 4715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1f0744884d44ecbd6b0e7a267a0cb7" candidate_uuid: "34e808b3b00042fcaf68279ed3c1133c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc"
I20260808 15:44:20.745213 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1f0744884d44ecbd6b0e7a267a0cb7" candidate_uuid: "34e808b3b00042fcaf68279ed3c1133c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762eb4a3028462d8c33cbab9f315251"
I20260808 15:44:20.745373 4715 raft_consensus.cc:3037] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:44:20.745417 4415 raft_consensus.cc:3037] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:44:20.747681 4715 raft_consensus.cc:2454] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e808b3b00042fcaf68279ed3c1133c in term 1.
I20260808 15:44:20.747756 4415 raft_consensus.cc:2454] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e808b3b00042fcaf68279ed3c1133c in term 1.
I20260808 15:44:20.748267 4504 leader_election.cc:302] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e808b3b00042fcaf68279ed3c1133c, 6762eb4a3028462d8c33cbab9f315251; no voters:
I20260808 15:44:20.748574 4799 raft_consensus.cc:2781] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:44:20.749142 4799 raft_consensus.cc:686] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 1 LEADER]: Becoming Leader. State: Replica: 34e808b3b00042fcaf68279ed3c1133c, State: Running, Role: LEADER
I20260808 15:44:20.749797 4799 consensus_queue.cc:237] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } }
I20260808 15:44:20.754921 4234 catalog_manager.cc:5495] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c reported cstate change: term changed from 0 to 1, leader changed from <none> to 34e808b3b00042fcaf68279ed3c1133c (127.23.130.2). New cstate: current_term: 1 leader_uuid: "34e808b3b00042fcaf68279ed3c1133c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e808b3b00042fcaf68279ed3c1133c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 46295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20260808 15:44:20.834048 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.828808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.23.130.2:46295)
0808 15:44:20.829031 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.829036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.829053 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:44:20.829175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.829178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.829262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.829395 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:44:20.829783 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.830547 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.831941 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.832343 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.832434 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.832908 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:44:20.833410 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:44:20.833544 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.833644 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.833712 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:44:20.842924 4715 raft_consensus.cc:1261] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 1 FOLLOWER]: Refusing update from remote peer 34e808b3b00042fcaf68279ed3c1133c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:44:20.842926 4415 raft_consensus.cc:1261] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 1 FOLLOWER]: Refusing update from remote peer 34e808b3b00042fcaf68279ed3c1133c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:44:20.843937 4802 consensus_queue.cc:1035] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6762eb4a3028462d8c33cbab9f315251" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:44:20.844338 4799 consensus_queue.cc:1035] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b3d6f31cee4192976521c0ddf3b4cc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:44:20.856747 4805 mvcc.cc:205] Tried to move back new op lower bound from 7316291014001336320 to 7316291013636427776. Current Snapshot: MvccSnapshot[applied={T|T < 7316291014001336320}]
I20260808 15:44:20.871798 4233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:44:20.872151 4233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:44:20.876298 4233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24011d2e21c84b1a8bd023b5d7667312: Sending DeleteTablet for 3 replicas of tablet db1f0744884d44ecbd6b0e7a267a0cb7
I20260808 15:44:20.877611 4693 tablet_service.cc:1514] Processing DeleteTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:44:20 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260808 15:44:20.877709 4550 tablet_service.cc:1514] Processing DeleteTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:44:20 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:38586
I20260808 15:44:20.877888 4393 tablet_service.cc:1514] Processing DeleteTablet for tablet db1f0744884d44ecbd6b0e7a267a0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:44:20 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:44562
I20260808 15:44:20.881433 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 4320
I20260808 15:44:20.888253 4819 tablet_replica.cc:330] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: stopping tablet replica
I20260808 15:44:20.889056 4819 raft_consensus.cc:2229] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:44:20.889830 4819 raft_consensus.cc:2258] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:44:20.890565 4817 tablet_replica.cc:330] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: stopping tablet replica
I20260808 15:44:20.891613 4817 raft_consensus.cc:2229] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:44:20.892411 4817 raft_consensus.cc:2258] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:44:20.893105 4818 tablet_replica.cc:330] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: stopping tablet replica
I20260808 15:44:20.893610 4818 raft_consensus.cc:2229] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:44:20.894486 4818 raft_consensus.cc:2258] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:44:20.895112 4819 ts_tablet_manager.cc:1905] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:44:20.897243 4817 ts_tablet_manager.cc:1905] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:44:20.900624 4818 ts_tablet_manager.cc:1905] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:44:20.902824 4819 ts_tablet_manager.cc:1918] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:44:20.903070 4819 log.cc:1196] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/db1f0744884d44ecbd6b0e7a267a0cb7
I20260808 15:44:20.903774 4819 ts_tablet_manager.cc:1939] T db1f0744884d44ecbd6b0e7a267a0cb7 P 6762eb4a3028462d8c33cbab9f315251: Deleting consensus metadata
I20260808 15:44:20.905596 4817 ts_tablet_manager.cc:1918] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:44:20.905800 4817 log.cc:1196] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/db1f0744884d44ecbd6b0e7a267a0cb7
I20260808 15:44:20.906036 4220 catalog_manager.cc:4841] TS 6762eb4a3028462d8c33cbab9f315251 (127.23.130.1:46697): tablet db1f0744884d44ecbd6b0e7a267a0cb7 (table test-table [id=a5643efcfb56441f854879bc66de6e49]) successfully deleted
I20260808 15:44:20.906373 4817 ts_tablet_manager.cc:1939] T db1f0744884d44ecbd6b0e7a267a0cb7 P a5b3d6f31cee4192976521c0ddf3b4cc: Deleting consensus metadata
I20260808 15:44:20.908156 4220 catalog_manager.cc:4841] TS a5b3d6f31cee4192976521c0ddf3b4cc (127.23.130.3:41695): tablet db1f0744884d44ecbd6b0e7a267a0cb7 (table test-table [id=a5643efcfb56441f854879bc66de6e49]) successfully deleted
I20260808 15:44:20.909833 4818 ts_tablet_manager.cc:1918] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:44:20.910022 4818 log.cc:1196] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/db1f0744884d44ecbd6b0e7a267a0cb7
I20260808 15:44:20.910672 4818 ts_tablet_manager.cc:1939] T db1f0744884d44ecbd6b0e7a267a0cb7 P 34e808b3b00042fcaf68279ed3c1133c: Deleting consensus metadata
Aug 08 15:44:20 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:44:20.912417 4220 catalog_manager.cc:4841] TS 34e808b3b00042fcaf68279ed3c1133c (127.23.130.2:46295): tablet db1f0744884d44ecbd6b0e7a267a0cb7 (table test-table [id=a5643efcfb56441f854879bc66de6e49]) successfully deleted
I20260808 15:44:20.920559 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:20.883546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.23.130.1:46697)
0808 15:44:20.883803 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:20.883813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:20.895423 (+ 11610us) server_negotiation.cc:407] Connection header received
0808 15:44:20.895775 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:20.895785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:20.895877 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:20.895963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:20.898199 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.899479 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.901810 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:20.902617 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:20.902727 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:20.913259 (+ 10532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:20.913274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:20.913289 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:20.913329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:20.917380 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.918158 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.918168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.918172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.918340 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:20.918805 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:20.918811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:20.918814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:20.919295 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:20.919611 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:20.919872 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 15:44:20.920141 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:44:20.921406 4460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:44:21.014195 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 4320
I20260808 15:44:21.036754 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 4474
Aug 08 15:44:21 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:44:21.054759 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:21.037590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.23.130.2:46295)
0808 15:44:21.037796 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:21.037801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:21.037899 (+ 98us) server_negotiation.cc:407] Connection header received
0808 15:44:21.038079 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:21.038083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:21.038159 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:21.038243 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:21.039975 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:21.040878 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:21.043005 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:21.043444 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:21.043541 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:21.048537 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:21.048550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:21.048567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:21.048603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:21.051888 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:21.052574 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:21.052582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:21.052585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:21.052733 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:21.053143 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:21.053148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:21.053150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:21.053509 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:21.053859 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:21.054124 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 15:44:21.054372 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:44:21.055433 4615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:44:21.171597 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 4474
I20260808 15:44:21.193357 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 4626
Aug 08 15:44:21 dist-test-slave-gp6n krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:44:21.209613 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:44:21.194095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.23.130.3:41695)
0808 15:44:21.194261 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 15:44:21.194265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:44:21.194283 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:44:21.194502 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:44:21.194506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:44:21.194572 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:44:21.194658 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:44:21.196055 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:21.197335 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:21.198688 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:44:21.199063 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:44:21.199152 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:44:21.203702 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:44:21.203716 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:44:21.203733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:44:21.203778 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:44:21.206849 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:21.207530 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:21.207537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:21.207539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:21.207684 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:44:21.208035 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:44:21.208039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:44:21.208041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:44:21.208420 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:44:21.208700 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 15:44:21.209029 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 15:44:21.209247 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:44:21.210330 4761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:44:21.299484 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 4626
I20260808 15:44:21.319115 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 4199
I20260808 15:44:21.320300 4266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:44:21.434242 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 4199
I20260808 15:44:21.460230 24072 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:44:21.805827 24072 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:44:21.806177 24072 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:44:22.151793 24072 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:44:22.152 UTC [3753] LOG: received smart shutdown request
2026-08-08 15:44:22.157 UTC [3753] LOG: background worker "logical replication launcher" (PID 3762) exited with exit code 1
2026-08-08 15:44:22.157 UTC [3757] LOG: shutting down
2026-08-08 15:44:22.332 UTC [3753] LOG: database system is shut down
2026-08-08T15:44:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50953 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:44:22 dist-test-slave-gp6n krb5kdc[4834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:44:22 dist-test-slave-gp6n krb5kdc[4834](info): set up 2 sockets
Aug 08 15:44:22 dist-test-slave-gp6n krb5kdc[4834](info): commencing operation
krb5kdc: starting...
W20260808 15:44:23.634115 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:44:23 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:44:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangeradmin_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangeradmin_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangerlookup_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangerlookup_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
I20260808 15:44:24.013406 24072 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2oCF2B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/postgres -l logfile start
2026-08-08 15:44:26.546 UTC [4870] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:44:26.546 UTC [4870] LOG: listening on IPv4 address "127.23.130.20", port 41473
2026-08-08 15:44:26.549 UTC [4870] LOG: listening on Unix socket "/tmp/.s.PGSQL.41473"
2026-08-08 15:44:26.573 UTC [4874] LOG: database system was shut down at 2026-08-08 15:44:25 UTC
2026-08-08 15:44:26.576 UTC [4870] LOG: database system is ready to accept connections
I20260808 15:44:27.661444 24072 mini_postgres.cc:96] Postgres bound to 41473
2026-08-08 15:44:27.684 UTC [4883] FATAL: database "slave" does not exist
127.23.130.20:41473 - accepting connections
I20260808 15:44:27.685381 24072 mini_ranger.cc:161] Starting Ranger...
I20260808 15:44:27.714753 24072 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:44:28.226619 24072 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:44:28.226750 24072 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin
2026-08-08 15:44:28,528 [I] DB FLAVOR :POSTGRES
2026-08-08 15:44:28,529 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:44:28,529 [I] Checking connection..
2026-08-08 15:44:28,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:44:28,788 [I] Checking connection passed.
2026-08-08 15:44:28,789 [I] --------- Verifying version history table ---------
2026-08-08 15:44:28,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:44:29,030 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:44:29,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:44:29,316 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:44:29,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:44:29,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:44:29.551 UTC [4949] WARNING: there is no transaction in progress
2026-08-08 15:44:29,559 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:44:29,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:44:29,814 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:44:29,814 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:44:29,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:44:30,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:44:30,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-08 15:44:30,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:44:30,822 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:44:30,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:44:31,088 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:44:31,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:44:31,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:44:31,618 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:44:31,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:44:32.621 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.637 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.653 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.670 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.686 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.706 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.760 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.767 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.781 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.792 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.803 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.817 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.827 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.834 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.842 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:32.850 UTC [5069] WARNING: there is no transaction in progress
2026-08-08 15:44:33,572 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:44:33,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:44:33,854 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:44:33,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:44:34,112 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:44:34,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:44:34,381 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:44:34,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:44:34,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:44:34,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-08 15:44:35,160 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:44:35,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:44:35,425 [I] DB_PATCHES have already been applied
I20260808 15:44:35.431321 24072 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2oCF2B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:44:35.431419 24072 mini_ranger.cc:193] Using host: 127.23.130.20
I20260808 15:44:35.452363 24072 mini_ranger.cc:227] Ranger admin URL: http://127.23.130.20:39203
Aug 08, 2026 3:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:44:36.693229 24072 mini_ranger.cc:160] Time spent starting Ranger: real 9.008s user 0.006s sys 0.052s
Aug 08, 2026 3:44:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:44:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 3:44:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:44:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:44:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:44:57.416116 24072 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangerkms_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/rangerkms_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.130.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.130.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.130.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/HTTP_127.23.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:44:57 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:44:57.612597 24072 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:44:57.636567 24072 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:44:58.520638 24072 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:44:58.520782 24072 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms
I20260808 15:44:58.520864 24072 mini_ranger_kms.cc:227] Using postgres at 127.23.130.20:41473
2026-08-08 15:44:58,891 [I] DB FLAVOR :POSTGRES
2026-08-08 15:44:58,892 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:44:58,892 [I] Checking connection
2026-08-08 15:44:58,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:44:59,127 [I] connection success
2026-08-08 15:44:59,127 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:44:59,127 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:44:59,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:44:59,394 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:44:59,395 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:44:59,395 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:44:59,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.130.20:41473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:44:59,657 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:45:00.121649 24072 mini_ranger_kms.cc:310] Created kms service
I20260808 15:45:00.341822 24072 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:45:00.389757 24072 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:45:00 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:00.795008 24072 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:45:00.795369 24072 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms:/tmp/dist-test-task2oCF2B/build/asan/bin/postgresql.jar:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2oCF2B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:45:00.795428 24072 mini_ranger_kms.cc:240] Using host: 127.23.130.20
I20260808 15:45:00.817845 24072 mini_ranger_kms.cc:276] Ranger KMS PID: 5271
I20260808 15:45:00.818039 24072 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.130.20:34791
Aug 08, 2026 3:45:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:45:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:45:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:45:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:45:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:45:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 15:45:03 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786203903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:03.264343 24072 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.652s user 0.091s sys 0.313s
I20260808 15:45:03.264598 24072 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:45:03.264833 24072 mini_ranger_kms.cc:401] 127.23.130.20:34791/kms/v1/keys
Aug 08 15:45:03 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
Aug 08 15:45:04 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.130.20@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
Aug 08 15:45:06 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:45:06.387738 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45163
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:45163
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:06.738694 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:06.739048 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:06.739224 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:06.746551 5324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:06.746701 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:06.746810 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:06.746918 5324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:06.747013 5324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:06.757422 5324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.130.20:34791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:45163
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:06.758826 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:06.760205 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:06.772856 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:06.772948 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:06.772980 5332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:06.773301 5324 server_base.cc:1034] running on GCE node
I20260808 15:45:06.774286 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:06.775698 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:06.776947 5324 hybrid_clock.cc:648] HybridClock initialized: now 1786203906776873 us; error 38 us; skew 500 ppm
I20260808 15:45:06.777438 5324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:06 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:06.780934 5324 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:06.782580 5324 webserver.cc:460] Webserver started at http://127.23.130.62:39137/ using document root <none> and password file <none>
I20260808 15:45:06.783208 5324 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:06.783314 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:06.783572 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:45:06 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:06.844099 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1"
format_stamp: "Formatted at 2026-08-08 15:45:06 on dist-test-slave-gp6n"
server_key: "7c87dc1a467523bf1dc43087b3b46323"
server_key_iv: "e20e916cda7822dec30f55b21f5109ff"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:06.845206 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1"
format_stamp: "Formatted at 2026-08-08 15:45:06 on dist-test-slave-gp6n"
server_key: "7c87dc1a467523bf1dc43087b3b46323"
server_key_iv: "e20e916cda7822dec30f55b21f5109ff"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:06.852304 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260808 15:45:06.873602 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:06.875492 5324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:45:06.875696 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1"
format_stamp: "Formatted at 2026-08-08 15:45:06 on dist-test-slave-gp6n"
server_key: "7c87dc1a467523bf1dc43087b3b46323"
server_key_iv: "e20e916cda7822dec30f55b21f5109ff"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:06.875968 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:06.940227 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:06.944664 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:06.945039 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:06.964730 5324 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:45163
I20260808 15:45:06.964762 5395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:45163 every 8 connection(s)
I20260808 15:45:06.966183 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:06.970118 5396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:06.970156 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5324
I20260808 15:45:06.970443 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:45:06.971186 24072 external_mini_cluster.cc:1428] Setting key 56adf6306c5f099537ee1aad999e4909
I20260808 15:45:06.984544 5396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Bootstrap starting.
Aug 08 15:45:06 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:06.987747 5396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:06.989136 5396 log.cc:824] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:06.992836 5396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: No bootstrap required, opened a new log
I20260808 15:45:06.997073 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:06.973318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.23.130.62:45163)
0808 15:45:06.974140 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:06.974155 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:06.974211 (+ 56us) server_negotiation.cc:407] Connection header received
0808 15:45:06.976856 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:06.976884 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:06.977414 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:06.977912 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:06.979254 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:06.981294 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:06.982810 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:06.983817 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:06.984022 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:06.988139 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:06.988151 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:06.988173 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:06.988231 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:06.992254 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:06.993035 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:06.993044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:06.993051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:06.993236 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:06.993679 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:06.993684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:06.993687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:06.994393 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:06.994731 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:06.995200 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 15:45:06.995623 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":298,"threads_started":1}
I20260808 15:45:07.002245 5396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:07.002673 5396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:07.002753 5396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd78fe7c3c84826a6f49d11a3ab37f1, State: Initialized, Role: FOLLOWER
I20260808 15:45:07.003286 5396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:07.003490 5396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:07.003587 5396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:07.003703 5396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:07.005769 5396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:07.006316 5396 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd78fe7c3c84826a6f49d11a3ab37f1; no voters:
I20260808 15:45:07.006989 5396 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:45:07.007241 5401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:07.008000 5401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd78fe7c3c84826a6f49d11a3ab37f1, State: Running, Role: LEADER
I20260808 15:45:07.008492 5401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:07.009138 5396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:07.011751 5402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } } }
I20260808 15:45:07.011785 5403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd78fe7c3c84826a6f49d11a3ab37f1. Latest consensus state: current_term: 1 leader_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } } }
I20260808 15:45:07.012176 5402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:07.012252 5403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:07.013823 5410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:07.018512 5410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:07.021649 5396 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:45:07.038753 5410 catalog_manager.cc:1347] Generated new cluster ID: 443815b8f7664bd0b793db17b1e3260a
I20260808 15:45:07.038923 5410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:07.062566 5410 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:45:07.064769 5410 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:07.079751 5410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Generated new TSK 0
I20260808 15:45:07.081004 5410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:45:07.841704 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:08 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:45:08.317802 5444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:08.318256 5444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:08.318460 5444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:08.330264 5444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:08.330469 5444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:08.330683 5444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:08.346693 5444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:08.348492 5444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:08.350333 5444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:08.370574 5454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:08.371568 5455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:08.380421 5444 server_base.cc:1034] running on GCE node
W20260808 15:45:08.380741 5458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:08.381731 5444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:08.383167 5444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:08.384456 5444 hybrid_clock.cc:648] HybridClock initialized: now 1786203908384383 us; error 57 us; skew 500 ppm
I20260808 15:45:08.384886 5444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:08 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:08.389616 5444 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:08.391959 5444 webserver.cc:460] Webserver started at http://127.23.130.1:37487/ using document root <none> and password file <none>
I20260808 15:45:08.392969 5444 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:08.393122 5444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:08.393514 5444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:45:08 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:08.427873 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "42979f560c2c47cebfe45b4847d57d42"
format_stamp: "Formatted at 2026-08-08 15:45:08 on dist-test-slave-gp6n"
server_key: "2a0f3adfbb4a7de6347e89c0bb2e9e31"
server_key_iv: "9a1ab54741f5b44b17ddf0cdbac13a8a"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:08.429240 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "42979f560c2c47cebfe45b4847d57d42"
format_stamp: "Formatted at 2026-08-08 15:45:08 on dist-test-slave-gp6n"
server_key: "2a0f3adfbb4a7de6347e89c0bb2e9e31"
server_key_iv: "9a1ab54741f5b44b17ddf0cdbac13a8a"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:08.447315 5444 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.010s sys 0.009s
I20260808 15:45:08.471077 5466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:08.472879 5444 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260808 15:45:08.473112 5444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "42979f560c2c47cebfe45b4847d57d42"
format_stamp: "Formatted at 2026-08-08 15:45:08 on dist-test-slave-gp6n"
server_key: "2a0f3adfbb4a7de6347e89c0bb2e9e31"
server_key_iv: "9a1ab54741f5b44b17ddf0cdbac13a8a"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:08.473395 5444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:08.489341 5444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:08.506774 5444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:08.507252 5444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:08.522962 5444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:08.547714 5444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:08.547860 5444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:08.548026 5444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:08.548112 5444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:45:08 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:08.778373 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:35487 every 8 connection(s)
I20260808 15:45:08.778429 5444 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:35487
I20260808 15:45:08.780292 5444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:08.781018 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5444
I20260808 15:45:08.781227 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:45:08.782030 24072 external_mini_cluster.cc:1428] Setting key 002510f5916057cc1e54a3ea9104b41b
Aug 08 15:45:08 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:08.849593 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:08.799744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.1:50467)
0808 15:45:08.820873 (+ 21129us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:08.820949 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:08.825252 (+ 4303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:08.826455 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:08.826470 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:08.827203 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:08.828098 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:08.828123 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:08.831826 (+ 3703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:08.831831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:08.833770 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:08.833786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:08.835109 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:08.835116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:08.835395 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:08.836257 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:08.836287 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:08.839061 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:08.843679 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:08.843690 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:08.843711 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:08.844394 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:08.844933 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:08.844939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:08.844943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:08.845221 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:08.846079 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:08.846088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:08.847023 (+ 935us) client_negotiation.cc:769] Sending connection context
0808 15:45:08.847365 (+ 342us) client_negotiation.cc:240] Negotiation successful
0808 15:45:08.847980 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20734,"thread_start_us":22964,"threads_started":1}
I20260808 15:45:08.850868 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:08.800501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:50467 (local address 127.23.130.62:45163)
0808 15:45:08.822779 (+ 22278us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:08.822786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:08.822813 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:45:08.825646 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:08.825653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:08.825756 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:08.825842 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:08.830510 (+ 4668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:08.831613 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:08.833951 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:08.834631 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:08.834736 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:08.839357 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:08.839367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:08.839372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:08.839430 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:08.843479 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:08.844582 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:08.844588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:08.844593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:08.844774 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:08.845374 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:08.845414 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:08.845418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:08.845816 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:08.845993 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:08.850173 (+ 4180us) server_negotiation.cc:299] Negotiation successful
0808 15:45:08.850480 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22141,"thread_start_us":23284,"threads_started":1}
I20260808 15:45:08.853176 5591 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:08.853540 5591 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:08.854511 5591 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:08.857182 5357 ts_manager.cc:194] Registered new tserver with Master: 42979f560c2c47cebfe45b4847d57d42 (127.23.130.1:35487)
I20260808 15:45:08.859297 5357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:50467
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
W20260808 15:45:08.907392 24072 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.130.2: real 0.122s user 0.004s sys 0.019s
I20260808 15:45:08.908351 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:09 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203908, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
W20260808 15:45:09.397050 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:09.397362 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:09.397462 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:09.404860 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:09.404982 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:09.405097 5598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:09.415298 5598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:09.416579 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:09.417912 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:09.431629 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:09.431658 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:09.431722 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:09.432765 5598 server_base.cc:1034] running on GCE node
I20260808 15:45:09.433456 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:09.434396 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:09.435691 5598 hybrid_clock.cc:648] HybridClock initialized: now 1786203909435647 us; error 36 us; skew 500 ppm
I20260808 15:45:09.436044 5598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:09 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203909, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:09.439428 5598 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:09.440891 5598 webserver.cc:460] Webserver started at http://127.23.130.2:43133/ using document root <none> and password file <none>
I20260808 15:45:09.441586 5598 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:09.441738 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:09.442126 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:45:09 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203909, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:09.458961 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "d6c951a338fe4174953866fddf3c7984"
format_stamp: "Formatted at 2026-08-08 15:45:09 on dist-test-slave-gp6n"
server_key: "94db33fdacae489045d4ab52ef6c0bb8"
server_key_iv: "d50a3da2b06e5c88f44d8710886f7eba"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:09.460115 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "d6c951a338fe4174953866fddf3c7984"
format_stamp: "Formatted at 2026-08-08 15:45:09 on dist-test-slave-gp6n"
server_key: "94db33fdacae489045d4ab52ef6c0bb8"
server_key_iv: "d50a3da2b06e5c88f44d8710886f7eba"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:09.467413 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 15:45:09.484246 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:09.485785 5598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:45:09.485968 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "d6c951a338fe4174953866fddf3c7984"
format_stamp: "Formatted at 2026-08-08 15:45:09 on dist-test-slave-gp6n"
server_key: "94db33fdacae489045d4ab52ef6c0bb8"
server_key_iv: "d50a3da2b06e5c88f44d8710886f7eba"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:09.486214 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:09.507066 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:09.511255 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:09.511560 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:09.512627 5598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:09.514119 5598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:09.514197 5598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:09.514360 5598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:09.514451 5598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 15:45:09.549561 5598 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:39329
I20260808 15:45:09.549614 5745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:39329 every 8 connection(s)
I20260808 15:45:09.550959 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:09.554095 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5598
I20260808 15:45:09.554311 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:45:09.554972 24072 external_mini_cluster.cc:1428] Setting key bef119d7868462ba6ffe8178c5462192
Aug 08 15:45:09 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203909, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:09.575088 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:09.553688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:47885 (local address 127.23.130.62:45163)
0808 15:45:09.554236 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:09.554243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:09.555352 (+ 1109us) server_negotiation.cc:407] Connection header received
0808 15:45:09.557702 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:09.557707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:09.557775 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:09.557864 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:09.559893 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:09.560600 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:09.563127 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:09.563529 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:09.563627 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:09.567935 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:09.567939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:09.567943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:09.567972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:09.571670 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:09.572381 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:09.572388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:09.572394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:09.572556 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:09.572995 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:09.573002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:09.573009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:09.573430 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:09.573583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:09.574256 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 15:45:09.574571 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
I20260808 15:45:09.575729 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:09.554209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.2:47885)
0808 15:45:09.555157 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:09.555216 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:09.557356 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:09.558166 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:09.558180 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:09.558854 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:09.559713 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:09.559742 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:09.560758 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:09.560766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:09.562986 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:09.563005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:09.563769 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:09.563779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:09.564114 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:09.564841 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:09.564875 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:09.567694 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:09.571815 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:09.571821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:09.571833 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:09.572250 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:09.572662 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:09.572665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:09.572667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:09.572861 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:09.573562 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:09.573568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:09.573866 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 15:45:09.574106 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 15:45:09.574570 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":380,"threads_started":1}
I20260808 15:45:09.577275 5746 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:09.577623 5746 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:09.578307 5746 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:09.579919 5357 ts_manager.cc:194] Registered new tserver with Master: d6c951a338fe4174953866fddf3c7984 (127.23.130.2:39329)
I20260808 15:45:09.580813 5357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:47885
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:45:09.620299 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:09.863102 5591 heartbeater.cc:503] Master 127.23.130.62:45163 was elected leader, sending a full tablet report...
W20260808 15:45:09.962493 5752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:09.962935 5752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:09.963114 5752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:09.970590 5752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:09.970728 5752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:09.970918 5752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:09.980887 5752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:09.982252 5752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:09.983582 5752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:09.997479 5761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:09.997388 5759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:09.997447 5758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:10.004163 5752 server_base.cc:1034] running on GCE node
I20260808 15:45:10.004889 5752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:10.005966 5752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:10.007200 5752 hybrid_clock.cc:648] HybridClock initialized: now 1786203910007137 us; error 58 us; skew 500 ppm
I20260808 15:45:10.007540 5752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203910, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:10.011101 5752 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:10.012735 5752 webserver.cc:460] Webserver started at http://127.23.130.3:38421/ using document root <none> and password file <none>
I20260808 15:45:10.013530 5752 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:10.013695 5752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:10.014041 5752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203910, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:10.035683 5752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "19e08962b14e44f1806e1d1da489b9c8"
format_stamp: "Formatted at 2026-08-08 15:45:10 on dist-test-slave-gp6n"
server_key: "a97d845f02f6a37ce8fe5a5384581b2c"
server_key_iv: "a2cabe0f40cb1ad1fe06d97fbf96eee9"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:10.036621 5752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "19e08962b14e44f1806e1d1da489b9c8"
format_stamp: "Formatted at 2026-08-08 15:45:10 on dist-test-slave-gp6n"
server_key: "a97d845f02f6a37ce8fe5a5384581b2c"
server_key_iv: "a2cabe0f40cb1ad1fe06d97fbf96eee9"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:10.042982 5752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 15:45:10.060971 5768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:10.062920 5752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:45:10.063138 5752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "19e08962b14e44f1806e1d1da489b9c8"
format_stamp: "Formatted at 2026-08-08 15:45:10 on dist-test-slave-gp6n"
server_key: "a97d845f02f6a37ce8fe5a5384581b2c"
server_key_iv: "a2cabe0f40cb1ad1fe06d97fbf96eee9"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:10.063421 5752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:10.083974 5752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:10.090257 5752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:10.090674 5752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:10.092065 5752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:10.094223 5752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:10.094334 5752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:10.094527 5752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:10.094621 5752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:10.132946 5752 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:42039
I20260808 15:45:10.132982 5892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:42039 every 8 connection(s)
I20260808 15:45:10.134279 5752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:10.138790 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5752
I20260808 15:45:10.138993 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:45:10.139494 24072 external_mini_cluster.cc:1428] Setting key 8357ae7528dc8956c2d47079ae723106
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203910, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:10.156018 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:10.137042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:36661 (local address 127.23.130.62:45163)
0808 15:45:10.137720 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:10.137731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:10.140735 (+ 3004us) server_negotiation.cc:407] Connection header received
0808 15:45:10.140786 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:10.140789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:10.140926 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:10.141043 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:10.143137 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.143863 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.145548 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.145986 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.146079 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:10.149847 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:10.149852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:10.149856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:10.149887 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:10.152465 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.153298 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.153302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.153304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.153493 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.154132 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.154138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.154141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.154498 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:10.154676 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:10.155583 (+ 907us) server_negotiation.cc:299] Negotiation successful
0808 15:45:10.155749 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":323,"threads_started":1}
I20260808 15:45:10.156961 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:10.137366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.3:36661)
0808 15:45:10.138170 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:10.138321 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:10.140410 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:10.141434 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:10.141447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:10.142070 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:10.142948 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:10.142984 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.144020 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.144024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:10.145422 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:10.145435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.146200 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.146205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:10.146554 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:10.147321 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:10.147345 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:10.149626 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:10.152600 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:10.152607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:10.152619 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:10.153147 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:10.153648 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:10.153656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:10.153659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:10.153966 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:10.154717 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:10.154724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:10.155122 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 15:45:10.155478 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 15:45:10.155899 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":337,"threads_started":1}
I20260808 15:45:10.158352 5893 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:10.158730 5893 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:10.159441 5893 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:10.160928 5357 ts_manager.cc:194] Registered new tserver with Master: 19e08962b14e44f1806e1d1da489b9c8 (127.23.130.3:42039)
I20260808 15:45:10.161748 5357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:36661
I20260808 15:45:10.167922 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:45:10.169672 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5444
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:45:10.187470 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:10.170731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.23.130.1:35487)
0808 15:45:10.171368 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:10.171396 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:10.171430 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:45:10.171485 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:10.171492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:10.171694 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:10.171898 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:10.173560 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.174682 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.176250 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.176814 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.176962 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:10.181553 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:10.181565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:10.181580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:10.181615 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:10.184522 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.185250 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.185256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.185259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.185413 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.185797 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.185803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.185806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.186152 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:10.186494 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:10.186816 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 15:45:10.187117 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":336,"threads_started":1}
I20260808 15:45:10.188791 5586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:10.278198 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5444
I20260808 15:45:10.297894 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5598
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:45:10.319940 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:10.298925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.23.130.2:39329)
0808 15:45:10.299540 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:10.299548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:10.299575 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:45:10.299639 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:10.299646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:10.299880 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:10.300037 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:10.301321 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.302577 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.303955 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.304656 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.304780 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:10.313580 (+ 8800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:10.313593 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:10.313607 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:10.313654 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:10.317000 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.317678 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.317686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.317688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.317852 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.318294 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.318298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.318300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.318729 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:10.319056 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:10.319298 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 15:45:10.319517 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":310,"threads_started":1}
I20260808 15:45:10.321218 5741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:10.404579 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5598
I20260808 15:45:10.423344 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5752
Aug 08 15:45:10 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:45:10.440786 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:10.424380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.23.130.3:42039)
0808 15:45:10.425079 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:10.425087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:10.425119 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:45:10.425187 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:10.425194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:10.425512 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:10.425672 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:10.426986 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.428131 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.429464 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:10.430040 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:10.430145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:10.434953 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:10.434964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:10.434978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:10.435012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:10.437890 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.438713 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.438720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.438722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.438930 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:10.439302 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:10.439306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:10.439308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:10.439672 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:10.439923 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:10.440188 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 15:45:10.440432 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":354,"threads_started":1}
I20260808 15:45:10.442034 5888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:10.531546 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5752
I20260808 15:45:10.550340 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5324
I20260808 15:45:10.551687 5391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:10.659822 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5324
I20260808 15:45:10.688505 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45163
--webserver_interface=127.23.130.62
--webserver_port=39137
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:45163
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:11.028410 5906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:11.028810 5906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:11.028982 5906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:11.036525 5906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:11.036685 5906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:11.036806 5906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:11.036911 5906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:11.037002 5906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:11.047650 5906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:45163
--ranger_config_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:45163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=39137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:11.049031 5906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:11.050526 5906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:11.063338 5911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:11.063395 5912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:11.063416 5914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:11.064185 5906 server_base.cc:1034] running on GCE node
I20260808 15:45:11.065040 5906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:11.066562 5906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:11.067880 5906 hybrid_clock.cc:648] HybridClock initialized: now 1786203911067820 us; error 53 us; skew 500 ppm
I20260808 15:45:11.068351 5906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:11 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:11.076594 5906 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:11.078156 5906 webserver.cc:460] Webserver started at http://127.23.130.62:39137/ using document root <none> and password file <none>
I20260808 15:45:11.078869 5906 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:11.078982 5906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:45:11 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:11.166029 5906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:45:11.169755 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:11.171490 5906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:45:11.171720 5906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1"
format_stamp: "Formatted at 2026-08-08 15:45:06 on dist-test-slave-gp6n"
server_key: "7c87dc1a467523bf1dc43087b3b46323"
server_key_iv: "e20e916cda7822dec30f55b21f5109ff"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:11.172843 5906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:11.185359 5906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:11.189103 5906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:11.189411 5906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:11.208127 5906 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:45163
I20260808 15:45:11.208148 5977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:45163 every 8 connection(s)
I20260808 15:45:11.209663 5906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:11.216540 5978 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:45:11.217561 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5906
I20260808 15:45:11.218506 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:35487
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:45163
--webserver_port=37487
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:11.219981 5978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Bootstrap starting.
I20260808 15:45:11.238977 5978 log.cc:824] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:11.248226 5978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:45:11.248804 5978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Bootstrap complete.
I20260808 15:45:11.258045 5978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:11.258988 5978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd78fe7c3c84826a6f49d11a3ab37f1, State: Initialized, Role: FOLLOWER
I20260808 15:45:11.259625 5978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:11.259869 5978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:11.260008 5978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:11.260186 5978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:45:11.263931 5978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:11.264423 5978 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd78fe7c3c84826a6f49d11a3ab37f1; no voters:
I20260808 15:45:11.265084 5978 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:45:11.265286 5982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:45:11.266585 5982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [term 2 LEADER]: Becoming Leader. State: Replica: 4dd78fe7c3c84826a6f49d11a3ab37f1, State: Running, Role: LEADER
I20260808 15:45:11.267205 5978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:11.267210 5982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } }
I20260808 15:45:11.270776 5984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd78fe7c3c84826a6f49d11a3ab37f1. Latest consensus state: current_term: 2 leader_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } } }
I20260808 15:45:11.271927 5984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:11.271564 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd78fe7c3c84826a6f49d11a3ab37f1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 45163 } } }
I20260808 15:45:11.272149 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:11.284044 5989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:11.291633 5989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:11.293557 5989 catalog_manager.cc:1259] Loaded cluster ID: 443815b8f7664bd0b793db17b1e3260a
I20260808 15:45:11.293754 5989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:11.299535 5989 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:11.299633 5978 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:45:11.301260 5989 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Loaded TSK: 0
I20260808 15:45:11.303282 5989 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:45:11.685042 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:11.685442 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:11.685596 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:11.697367 5980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:11.697508 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:11.697701 5980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:11.713737 5980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:35487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=37487
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:11.715541 5980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:11.717396 5980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:11.770529 6018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:11.773414 6021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:11.773475 6019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:11.775619 5980 server_base.cc:1034] running on GCE node
I20260808 15:45:11.776865 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:11.779487 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:11.785669 5980 hybrid_clock.cc:648] HybridClock initialized: now 1786203911785585 us; error 60 us; skew 500 ppm
I20260808 15:45:11.786191 5980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:11 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:11.793239 5980 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:11.830914 5980 webserver.cc:460] Webserver started at http://127.23.130.1:37487/ using document root <none> and password file <none>
I20260808 15:45:11.833211 5980 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:11.833487 5980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:45:11 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:11.881642 5980 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.003s sys 0.012s
I20260808 15:45:11.888388 6031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:11.890458 5980 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260808 15:45:11.890661 5980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "42979f560c2c47cebfe45b4847d57d42"
format_stamp: "Formatted at 2026-08-08 15:45:08 on dist-test-slave-gp6n"
server_key: "2a0f3adfbb4a7de6347e89c0bb2e9e31"
server_key_iv: "9a1ab54741f5b44b17ddf0cdbac13a8a"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:11.892339 5980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:11.914696 5980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:11.922060 5980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:11.922443 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:11.923691 5980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:11.925787 5980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:11.925937 5980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:11.926090 5980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:11.926195 5980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:11.980087 5980 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:35487
I20260808 15:45:11.980290 6152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:35487 every 8 connection(s)
I20260808 15:45:11.982199 5980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:11.983646 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 5980
I20260808 15:45:11.984599 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:39329
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:45163
--webserver_port=43133
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:12.035136 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:11.986810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.1:47379)
0808 15:45:11.999637 (+ 12827us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:11.999710 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:12.002987 (+ 3277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:12.006916 (+ 3929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:12.006932 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:12.007712 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:12.008642 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:12.008679 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.011537 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.011544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:12.014008 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:12.014026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.016201 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.016211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:12.016527 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:12.017485 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:12.017527 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:12.020834 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:12.028845 (+ 8011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:12.028855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:12.028873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:12.029592 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:12.030230 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:12.030235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:12.030239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:12.030586 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:12.031886 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:12.031894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:12.032366 (+ 472us) client_negotiation.cc:769] Sending connection context
0808 15:45:12.032704 (+ 338us) client_negotiation.cc:240] Negotiation successful
0808 15:45:12.033485 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9620,"thread_start_us":11004,"threads_started":1}
I20260808 15:45:12.038074 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:11.986747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:47379 (local address 127.23.130.62:45163)
0808 15:45:12.003906 (+ 17159us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:12.003929 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:12.003986 (+ 57us) server_negotiation.cc:407] Connection header received
0808 15:45:12.004488 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:12.004515 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:12.005041 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:12.005498 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:12.008931 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.011282 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.014270 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.015529 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.015731 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:12.021200 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:12.021214 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:12.021236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:12.021289 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:12.028634 (+ 7345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:12.029804 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:12.029815 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:12.029825 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:12.030066 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:12.030748 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:12.030754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:12.030757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:12.031568 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:12.031783 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:12.036134 (+ 4351us) server_negotiation.cc:299] Negotiation successful
0808 15:45:12.036634 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16851,"thread_start_us":16597,"threads_started":1}
I20260808 15:45:12.042646 6153 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:12.042995 6153 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:12.044037 6153 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
W20260808 15:45:12.047981 6153 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:45163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:45:12.048228 6153 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:12.048550 6153 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:12.050087 6153 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:12.050509 6153 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
W20260808 15:45:12.051702 6153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:45:12.420254 6157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:12.420660 6157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:12.420840 6157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:12.433782 6157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:12.434029 6157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:12.434291 6157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:12.452046 6157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:39329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=43133
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:12.454080 6157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:12.456025 6157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:12.471902 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:12.481387 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:12.495767 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:12.496816 6157 server_base.cc:1034] running on GCE node
I20260808 15:45:12.497651 6157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:12.499054 6157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:12.500357 6157 hybrid_clock.cc:648] HybridClock initialized: now 1786203912500258 us; error 87 us; skew 500 ppm
I20260808 15:45:12.500784 6157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:12.505178 6157 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:12.507896 6157 webserver.cc:460] Webserver started at http://127.23.130.2:43133/ using document root <none> and password file <none>
I20260808 15:45:12.508783 6157 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:12.508945 6157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:12.535569 6157 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:45:12.539682 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:12.541604 6157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:45:12.541846 6157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "d6c951a338fe4174953866fddf3c7984"
format_stamp: "Formatted at 2026-08-08 15:45:09 on dist-test-slave-gp6n"
server_key: "94db33fdacae489045d4ab52ef6c0bb8"
server_key_iv: "d50a3da2b06e5c88f44d8710886f7eba"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:12.543325 6157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:12.569679 6157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:12.575250 6157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:12.575628 6157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:12.582942 6157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:12.594270 6157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:12.594426 6157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:12.594607 6157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:12.594696 6157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:12.651436 6157 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:39329
I20260808 15:45:12.651521 6305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:39329 every 8 connection(s)
I20260808 15:45:12.653414 6157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:12.660655 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6157
I20260808 15:45:12.661490 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.130.20:34791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:42039
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:45163
--webserver_port=38421
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:12.683660 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:12.663191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:40445 (local address 127.23.130.62:45163)
0808 15:45:12.664755 (+ 1564us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:12.664760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:12.664777 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:45:12.664831 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:12.664836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:12.664903 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:12.665034 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:12.667361 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.668538 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.670130 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.670655 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.670797 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:12.674785 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:12.674795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:12.674803 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:12.674858 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:12.679031 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:12.680141 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:12.680148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:12.680152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:12.680411 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:12.681259 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:12.681267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:12.681274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:12.681704 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:12.681876 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:12.682886 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0808 15:45:12.683158 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":1296,"threads_started":1}
I20260808 15:45:12.684991 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:12.659772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.2:40445)
0808 15:45:12.660586 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:12.660640 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:12.664040 (+ 3400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:12.665325 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:12.665342 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:12.666125 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:12.667095 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:12.667123 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.668706 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.668710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:12.669966 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:12.669985 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:12.670890 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:12.670897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:12.671128 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:12.671827 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:12.671850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:12.674533 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:12.679202 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:12.679213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:12.679231 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:12.679929 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:12.680671 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:12.680678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:12.680711 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:12.681128 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:12.681880 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:12.681889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:12.682332 (+ 443us) client_negotiation.cc:769] Sending connection context
0808 15:45:12.682708 (+ 376us) client_negotiation.cc:240] Negotiation successful
0808 15:45:12.683316 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":3758,"threads_started":1}
I20260808 15:45:12.687024 6306 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:12.687402 6306 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:12.688370 6306 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:12.690735 5940 ts_manager.cc:194] Registered new tserver with Master: d6c951a338fe4174953866fddf3c7984 (127.23.130.2:39329)
I20260808 15:45:12.693261 5940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:40445
Aug 08 15:45:12 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:13.051858 6153 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:13.052309 6153 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:13.053609 5940 ts_manager.cc:194] Registered new tserver with Master: 42979f560c2c47cebfe45b4847d57d42 (127.23.130.1:35487)
I20260808 15:45:13.055024 5940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:47379
W20260808 15:45:13.123158 6311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:13.123567 6311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:13.123781 6311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:13.135048 6311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:13.135193 6311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:13.135433 6311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:13.150633 6311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.130.20:34791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:42039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=38421
--tserver_master_addrs=127.23.130.62:45163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:13.152275 6311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:13.153993 6311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:13.171527 6317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:13.173749 6318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:13.175653 6311 server_base.cc:1034] running on GCE node
W20260808 15:45:13.176388 6320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:13.177758 6311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:13.179083 6311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:13.180389 6311 hybrid_clock.cc:648] HybridClock initialized: now 1786203913180323 us; error 45 us; skew 500 ppm
I20260808 15:45:13.180771 6311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:13 dist-test-slave-gp6n krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:13.185345 6311 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:13.192021 6311 webserver.cc:460] Webserver started at http://127.23.130.3:38421/ using document root <none> and password file <none>
I20260808 15:45:13.193046 6311 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:13.193215 6311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:45:13 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:13.252748 6311 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.000s sys 0.011s
Aug 08 15:45:13 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for HTTP/127.23.130.20@KRBTEST.COM
I20260808 15:45:13.262871 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:13.264921 6311 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.003s
I20260808 15:45:13.265162 6311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "19e08962b14e44f1806e1d1da489b9c8"
format_stamp: "Formatted at 2026-08-08 15:45:10 on dist-test-slave-gp6n"
server_key: "a97d845f02f6a37ce8fe5a5384581b2c"
server_key_iv: "a2cabe0f40cb1ad1fe06d97fbf96eee9"
server_key_version: "kuduclusterkey@0"
I20260808 15:45:13.266723 6311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:13.346787 6311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:13.356261 6311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:13.356647 6311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:13.377569 6311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:13.387346 6311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:13.387562 6311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 15:45:13.387729 6311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:13.387840 6311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:13.639169 6311 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:42039
I20260808 15:45:13.639230 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:42039 every 8 connection(s)
I20260808 15:45:13.641173 6311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:13.642299 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6311
I20260808 15:45:13.698395 6306 heartbeater.cc:503] Master 127.23.130.62:45163 was elected leader, sending a full tablet report...
Aug 08 15:45:13 dist-test-slave-gp6n krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:13.733990 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.643823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.23.130.62:45163)
0808 15:45:13.701111 (+ 57288us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.701118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.709622 (+ 8504us) server_negotiation.cc:407] Connection header received
0808 15:45:13.712175 (+ 2553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.712181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.712255 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.712331 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:13.718516 (+ 6185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.719716 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.722391 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.723229 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.723337 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.727180 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:13.727187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:13.727193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:13.727234 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:13.730863 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.731758 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.731764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.731767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.731922 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.732374 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.732379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.732381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.732661 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:13.732981 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.733254 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.733484 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":57154,"thread_start_us":63682,"threads_started":1}
I20260808 15:45:13.735100 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.646365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35493 (local address 127.23.130.62:45163)
0808 15:45:13.707064 (+ 60699us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.707070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.707091 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:45:13.707158 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.707165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.707254 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.707365 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:13.714868 (+ 7503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.716056 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.722163 (+ 6107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.722935 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.723070 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.727551 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:13.727558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:13.727562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:13.727612 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:13.731323 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.732209 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.732214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.732218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.732340 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.732893 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.732898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.732901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.733230 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:13.733346 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.734517 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.734732 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":60573,"thread_start_us":54735,"threads_started":1}
I20260808 15:45:13.736140 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.645751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:45163 (local address 127.23.130.3:35493)
0808 15:45:13.697232 (+ 51481us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.697301 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.700209 (+ 2908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.710000 (+ 9791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.710015 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.710789 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:13.711703 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.711732 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.716289 (+ 4557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.716294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.721967 (+ 5673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.721983 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.723402 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.723409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.723717 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.724445 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:13.724476 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:13.727228 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:13.731480 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:13.731490 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:13.731510 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:13.732086 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:13.732477 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:13.732482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:13.732485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:13.732757 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:13.733426 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:13.733446 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:13.733862 (+ 416us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.734152 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.734740 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51058,"thread_start_us":55347,"threads_started":1}
I20260808 15:45:13.737648 6463 heartbeater.cc:347] Connected to a master server at 127.23.130.62:45163
I20260808 15:45:13.737969 6463 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:13.738848 6463 heartbeater.cc:511] Master 127.23.130.62:45163 requested a full tablet report, sending...
I20260808 15:45:13.740304 5940 ts_manager.cc:194] Registered new tserver with Master: 19e08962b14e44f1806e1d1da489b9c8 (127.23.130.3:42039)
I20260808 15:45:13.741097 5940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:35493
I20260808 15:45:13.749544 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:45:13.768116 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.753229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.23.130.62:45163)
0808 15:45:13.753486 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.753496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.753922 (+ 426us) server_negotiation.cc:407] Connection header received
0808 15:45:13.754085 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.754090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.754152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.754233 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:13.756320 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.757034 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.758969 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.759385 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.759498 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.762646 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:13.762651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:13.762654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:13.762686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:13.765353 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.766122 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.766126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.766129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.766252 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:13.766663 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:13.766667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:13.766668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:13.766929 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:13.767120 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.767463 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.767700 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:45:13.773036 5940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:45:13.780740 5940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:45:13.823391 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.795736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:39329 (local address 127.0.0.1:55872)
0808 15:45:13.803157 (+ 7421us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.803218 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.803461 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.811730 (+ 8269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.811739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.811775 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:13.812235 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.812248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.814466 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.814472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.816857 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.816873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.822310 (+ 5437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.822317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.822571 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.822635 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.822754 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.822992 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7069,"threads_started":1}
I20260808 15:45:13.827781 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.796000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.23.130.2:39329)
0808 15:45:13.810985 (+ 14985us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.810997 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.811034 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:45:13.811161 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.811171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.811466 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.812474 (+ 1008us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:13.812729 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.814280 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.820412 (+ 6132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.822041 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.826858 (+ 4817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.827127 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.827214 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.827401 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14785,"thread_start_us":14619,"threads_started":1}
I20260808 15:45:13.831795 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.805851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.23.130.1:35487)
0808 15:45:13.812017 (+ 6166us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.812026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.824023 (+ 11997us) server_negotiation.cc:407] Connection header received
0808 15:45:13.824324 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.824332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.824575 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.824742 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:13.825428 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.827027 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.829587 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.830771 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.830938 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.831167 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.831311 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.831469 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5990,"thread_start_us":5812,"threads_started":1}
I20260808 15:45:13.831794 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.805022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35487 (local address 127.0.0.1:34028)
0808 15:45:13.823889 (+ 18867us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.823909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.824128 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.825043 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.825048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.825072 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:13.825239 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.825248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.827210 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.827215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.829419 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.829434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.830947 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.830953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.831126 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.831164 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.831265 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.831409 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18744,"mutex_wait_us":52,"thread_start_us":6010,"threads_started":1}
I20260808 15:45:13.832615 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.796179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:42039 (local address 127.0.0.1:56880)
0808 15:45:13.811300 (+ 15121us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.811320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.811453 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.822531 (+ 11078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.822539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.822579 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:13.823180 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.823198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.827690 (+ 4492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.827699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.829868 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.829882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.831780 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.831787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.831976 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.832008 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.832076 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.832201 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14999,"mutex_wait_us":4,"thread_start_us":6941,"threads_started":1}
I20260808 15:45:13.833032 6237 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 (DEFAULT_TABLE table=test-table [id=52c3b5a740454043ac9a656545c6554b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:13.833678 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.797012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.23.130.3:42039)
0808 15:45:13.821402 (+ 24390us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.821416 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.821457 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:45:13.821589 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.821602 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.821961 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.823473 (+ 1512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:13.823878 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.827476 (+ 3598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.830043 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.831519 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.832812 (+ 1293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.833032 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.833105 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.833324 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24125,"thread_start_us":25143,"threads_started":1}
I20260808 15:45:13.834573 6237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb7a36a5df49fa96b806434485c2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:13.835564 6083 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 (DEFAULT_TABLE table=test-table [id=52c3b5a740454043ac9a656545c6554b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:13.836676 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb7a36a5df49fa96b806434485c2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:13.837229 6389 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 (DEFAULT_TABLE table=test-table [id=52c3b5a740454043ac9a656545c6554b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:13.838595 6389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb7a36a5df49fa96b806434485c2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:13.848205 6485 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Bootstrap starting.
I20260808 15:45:13.850286 6486 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Bootstrap starting.
I20260808 15:45:13.852694 6485 tablet_bootstrap.cc:654] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:13.853978 6486 tablet_bootstrap.cc:654] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:13.854151 6485 log.cc:824] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:13.855125 6487 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Bootstrap starting.
I20260808 15:45:13.855384 6486 log.cc:824] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:13.858551 6486 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: No bootstrap required, opened a new log
I20260808 15:45:13.858825 6486 ts_tablet_manager.cc:1397] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:45:13.858971 6487 tablet_bootstrap.cc:654] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:13.859172 6485 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: No bootstrap required, opened a new log
I20260808 15:45:13.859457 6485 ts_tablet_manager.cc:1397] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260808 15:45:13.860224 6487 log.cc:824] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:13.863261 6487 tablet_bootstrap.cc:492] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: No bootstrap required, opened a new log
I20260808 15:45:13.863507 6487 ts_tablet_manager.cc:1397] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260808 15:45:13.867297 6486 raft_consensus.cc:348] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.867674 6486 raft_consensus.cc:374] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:13.867851 6486 raft_consensus.cc:729] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c951a338fe4174953866fddf3c7984, State: Initialized, Role: FOLLOWER
I20260808 15:45:13.868463 6486 consensus_queue.cc:260] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.869774 6485 raft_consensus.cc:348] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.870208 6485 raft_consensus.cc:374] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:13.870309 6485 raft_consensus.cc:729] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42979f560c2c47cebfe45b4847d57d42, State: Initialized, Role: FOLLOWER
I20260808 15:45:13.870913 6485 consensus_queue.cc:260] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.871698 6486 ts_tablet_manager.cc:1428] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260808 15:45:13.872577 6153 heartbeater.cc:503] Master 127.23.130.62:45163 was elected leader, sending a full tablet report...
I20260808 15:45:13.873159 6485 ts_tablet_manager.cc:1428] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260808 15:45:13.874125 6487 raft_consensus.cc:348] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.874608 6487 raft_consensus.cc:374] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:13.874722 6487 raft_consensus.cc:729] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e08962b14e44f1806e1d1da489b9c8, State: Initialized, Role: FOLLOWER
I20260808 15:45:13.875324 6487 consensus_queue.cc:260] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.876484 6463 heartbeater.cc:503] Master 127.23.130.62:45163 was elected leader, sending a full tablet report...
I20260808 15:45:13.877149 6487 ts_tablet_manager.cc:1428] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260808 15:45:13.917192 6494 raft_consensus.cc:482] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:45:13.917541 6494 raft_consensus.cc:504] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.919440 6494 leader_election.cc:288] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c951a338fe4174953866fddf3c7984 (127.23.130.2:39329), 42979f560c2c47cebfe45b4847d57d42 (127.23.130.1:35487)
W20260808 15:45:13.919799 6307 tablet.cc:2369] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:13.926883 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.919710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:39329 (local address 127.23.130.3:38361)
0808 15:45:13.920222 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.920244 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.920402 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.921030 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.921041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.921071 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:13.921307 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.921324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.922640 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.922645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.924611 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.924627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.926176 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.926180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.926352 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.926379 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.926519 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.926622 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260808 15:45:13.926957 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.920044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:38361 (local address 127.23.130.2:39329)
0808 15:45:13.920247 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.920254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.920528 (+ 274us) server_negotiation.cc:407] Connection header received
0808 15:45:13.920619 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.920624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.920728 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.920829 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:13.921476 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.922471 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.924795 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.925981 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.926148 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.926225 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.926568 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.926705 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:45:13.927670 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.920414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:36331 (local address 127.23.130.1:35487)
0808 15:45:13.920627 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.920635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.921403 (+ 768us) server_negotiation.cc:407] Connection header received
0808 15:45:13.921533 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.921539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.921633 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.921742 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:13.922270 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.923082 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.925453 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.926690 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.926846 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.926911 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.927271 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.927418 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:45:13.927695 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.919968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:35487 (local address 127.23.130.3:36331)
0808 15:45:13.921272 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:13.921290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:13.921416 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:13.921761 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:13.921764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:13.921785 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:13.922110 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.922118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.923239 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.923245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.925279 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:13.925293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.926921 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.926928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:13.927110 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.927148 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:45:13.927232 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:45:13.927351 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":913,"threads_started":1}
I20260808 15:45:13.927953 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb7a36a5df49fa96b806434485c2c1" candidate_uuid: "19e08962b14e44f1806e1d1da489b9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c951a338fe4174953866fddf3c7984" is_pre_election: true
I20260808 15:45:13.928579 6257 raft_consensus.cc:2454] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e08962b14e44f1806e1d1da489b9c8 in term 0.
I20260808 15:45:13.928656 6105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb7a36a5df49fa96b806434485c2c1" candidate_uuid: "19e08962b14e44f1806e1d1da489b9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42979f560c2c47cebfe45b4847d57d42" is_pre_election: true
I20260808 15:45:13.929193 6105 raft_consensus.cc:2454] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e08962b14e44f1806e1d1da489b9c8 in term 0.
I20260808 15:45:13.929339 6333 leader_election.cc:302] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e08962b14e44f1806e1d1da489b9c8, d6c951a338fe4174953866fddf3c7984; no voters:
I20260808 15:45:13.929826 6494 raft_consensus.cc:2781] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:45:13.929991 6494 raft_consensus.cc:482] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:45:13.930119 6494 raft_consensus.cc:3037] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:13.932391 6494 raft_consensus.cc:504] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.932973 6494 leader_election.cc:288] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [CANDIDATE]: Term 1 election: Requested vote from peers d6c951a338fe4174953866fddf3c7984 (127.23.130.2:39329), 42979f560c2c47cebfe45b4847d57d42 (127.23.130.1:35487)
I20260808 15:45:13.933501 6105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb7a36a5df49fa96b806434485c2c1" candidate_uuid: "19e08962b14e44f1806e1d1da489b9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42979f560c2c47cebfe45b4847d57d42"
I20260808 15:45:13.933501 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb7a36a5df49fa96b806434485c2c1" candidate_uuid: "19e08962b14e44f1806e1d1da489b9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c951a338fe4174953866fddf3c7984"
I20260808 15:45:13.933715 6257 raft_consensus.cc:3037] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:13.933710 6105 raft_consensus.cc:3037] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:13.936976 6105 raft_consensus.cc:2454] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e08962b14e44f1806e1d1da489b9c8 in term 1.
I20260808 15:45:13.936983 6257 raft_consensus.cc:2454] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e08962b14e44f1806e1d1da489b9c8 in term 1.
I20260808 15:45:13.937372 6333 leader_election.cc:302] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e08962b14e44f1806e1d1da489b9c8, d6c951a338fe4174953866fddf3c7984; no voters:
I20260808 15:45:13.937721 6494 raft_consensus.cc:2781] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:13.938182 6494 raft_consensus.cc:686] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 19e08962b14e44f1806e1d1da489b9c8, State: Running, Role: LEADER
I20260808 15:45:13.938709 6494 consensus_queue.cc:237] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } }
I20260808 15:45:13.943420 5939 catalog_manager.cc:5495] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e08962b14e44f1806e1d1da489b9c8 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "19e08962b14e44f1806e1d1da489b9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e08962b14e44f1806e1d1da489b9c8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 } health_report { overall_health: UNKNOWN } } }
I20260808 15:45:13.966658 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:13.961508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.23.130.3:42039)
0808 15:45:13.961710 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:13.961715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:13.961733 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:45:13.961900 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:13.961903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:13.961999 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:13.962132 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:45:13.962514 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.963238 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.964612 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:13.965023 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:13.965123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:13.965583 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:45:13.966107 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:45:13.966221 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:13.966333 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 15:45:13.966398 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 15:45:13.972594 6466 tablet.cc:2369] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:13.975198 6105 raft_consensus.cc:1261] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 1 FOLLOWER]: Refusing update from remote peer 19e08962b14e44f1806e1d1da489b9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:13.975215 6257 raft_consensus.cc:1261] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 1 FOLLOWER]: Refusing update from remote peer 19e08962b14e44f1806e1d1da489b9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:13.976215 6496 consensus_queue.cc:1035] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c951a338fe4174953866fddf3c7984" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:13.976576 6494 consensus_queue.cc:1035] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42979f560c2c47cebfe45b4847d57d42" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:13.991531 6499 mvcc.cc:205] Tried to move back new op lower bound from 7316291231632052224 to 7316291231496577024. Current Snapshot: MvccSnapshot[applied={T|T < 7316291231632052224}]
I20260808 15:45:14.006656 5939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:14.006958 5939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:14.010800 5939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dd78fe7c3c84826a6f49d11a3ab37f1: Sending DeleteTablet for 3 replicas of tablet 45eb7a36a5df49fa96b806434485c2c1
I20260808 15:45:14.011966 6389 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:56880
I20260808 15:45:14.011979 6237 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:55872
I20260808 15:45:14.011981 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb7a36a5df49fa96b806434485c2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:34028
I20260808 15:45:14.015363 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5980
I20260808 15:45:14.024552 6511 tablet_replica.cc:330] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: stopping tablet replica
I20260808 15:45:14.025179 6511 raft_consensus.cc:2229] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:45:14.026000 6511 raft_consensus.cc:2258] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:14.027802 6512 tablet_replica.cc:330] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: stopping tablet replica
I20260808 15:45:14.028254 6512 raft_consensus.cc:2229] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:14.028807 6512 raft_consensus.cc:2258] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:14.029220 6511 ts_tablet_manager.cc:1905] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:14.030931 6512 ts_tablet_manager.cc:1905] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:14.036526 6511 ts_tablet_manager.cc:1918] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:14.036705 6511 log.cc:1196] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/45eb7a36a5df49fa96b806434485c2c1
I20260808 15:45:14.037276 6511 ts_tablet_manager.cc:1939] T 45eb7a36a5df49fa96b806434485c2c1 P 19e08962b14e44f1806e1d1da489b9c8: Deleting consensus metadata
I20260808 15:45:14.038388 6512 ts_tablet_manager.cc:1918] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:14.038619 6512 log.cc:1196] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/45eb7a36a5df49fa96b806434485c2c1
I20260808 15:45:14.039223 6512 ts_tablet_manager.cc:1939] T 45eb7a36a5df49fa96b806434485c2c1 P 42979f560c2c47cebfe45b4847d57d42: Deleting consensus metadata
I20260808 15:45:14.039234 5924 catalog_manager.cc:4841] TS 19e08962b14e44f1806e1d1da489b9c8 (127.23.130.3:42039): tablet 45eb7a36a5df49fa96b806434485c2c1 (table test-table [id=52c3b5a740454043ac9a656545c6554b]) successfully deleted
I20260808 15:45:14.040233 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:14.016659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.23.130.1:35487)
0808 15:45:14.016839 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:14.016845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:14.016865 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:45:14.017033 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:14.017039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:14.017111 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:14.017204 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:14.025118 (+ 7914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.026376 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.028561 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.029158 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.029327 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:14.032208 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:14.032219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:14.032235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:14.032266 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:14.036325 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.037170 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.037183 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.037189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.037525 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.037918 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.037925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.037931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.038667 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:14.039057 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:14.039514 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 15:45:14.039772 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:45:14.040218 6510 tablet_replica.cc:330] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: stopping tablet replica
I20260808 15:45:14.040985 6147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:14.040951 5926 catalog_manager.cc:4841] TS 42979f560c2c47cebfe45b4847d57d42 (127.23.130.1:35487): tablet 45eb7a36a5df49fa96b806434485c2c1 (table test-table [id=52c3b5a740454043ac9a656545c6554b]) successfully deleted
I20260808 15:45:14.041164 6510 raft_consensus.cc:2229] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:14.041814 6510 raft_consensus.cc:2258] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:14.044189 6510 ts_tablet_manager.cc:1905] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:14.051930 6510 ts_tablet_manager.cc:1918] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:14.052120 6510 log.cc:1196] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/45eb7a36a5df49fa96b806434485c2c1
I20260808 15:45:14.052595 6510 ts_tablet_manager.cc:1939] T 45eb7a36a5df49fa96b806434485c2c1 P d6c951a338fe4174953866fddf3c7984: Deleting consensus metadata
I20260808 15:45:14.054155 5925 catalog_manager.cc:4841] TS d6c951a338fe4174953866fddf3c7984 (127.23.130.2:39329): tablet 45eb7a36a5df49fa96b806434485c2c1 (table test-table [id=52c3b5a740454043ac9a656545c6554b]) successfully deleted
I20260808 15:45:14.149474 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5980
I20260808 15:45:14.170153 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6157
I20260808 15:45:14.188016 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:14.171064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.23.130.2:39329)
0808 15:45:14.171254 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:14.171260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:14.171281 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:45:14.171502 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:14.171508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:14.171606 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:14.171762 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:14.173316 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.174649 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.176081 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.176739 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.176873 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:14.180349 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:14.180364 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:14.180381 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:14.180424 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:14.183747 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.184789 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.184798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.184800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.185162 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.185576 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.185581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.185584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.186245 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:14.186740 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:14.187077 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 15:45:14.187451 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:45:14.188831 6301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:14.298151 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6157
I20260808 15:45:14.318041 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6311
I20260808 15:45:14.335862 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:14.319005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.23.130.3:42039)
0808 15:45:14.319199 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:14.319203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:14.319321 (+ 118us) server_negotiation.cc:407] Connection header received
0808 15:45:14.319514 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:14.319518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:14.319580 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:14.319670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:14.321672 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.322571 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.324606 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:14.325294 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:14.325426 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:14.328309 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:14.328323 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:14.328342 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:14.328395 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:14.332704 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.333444 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.333455 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.333459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.333644 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:14.334121 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:14.334126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:14.334131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:14.334663 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:14.335028 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:14.335226 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 15:45:14.335455 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:45:14.336774 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:14.458002 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6311
I20260808 15:45:14.480535 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 5906
I20260808 15:45:14.481739 5972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:14.595572 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 5906
I20260808 15:45:14.623948 24072 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:45:14.969228 24072 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:45:14.969537 24072 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:45:15.314754 24072 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:45:15.315 UTC [4870] LOG: received smart shutdown request
2026-08-08 15:45:15.318 UTC [4870] LOG: background worker "logical replication launcher" (PID 4881) exited with exit code 1
2026-08-08 15:45:15.318 UTC [4876] LOG: shutting down
2026-08-08 15:45:15.485 UTC [4870] LOG: database system is shut down
2026-08-08T15:45:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53156 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:45:15 dist-test-slave-gp6n krb5kdc[6526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:45:15 dist-test-slave-gp6n krb5kdc[6526](info): set up 2 sockets
Aug 08 15:45:15 dist-test-slave-gp6n krb5kdc[6526](info): commencing operation
krb5kdc: starting...
W20260808 15:45:16.813131 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:45:16 dist-test-slave-gp6n krb5kdc[6526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:45:17.105901 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38255
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:17.448782 6540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:17.449090 6540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:17.449179 6540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:17.456593 6540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:17.456694 6540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:17.456740 6540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:17.456799 6540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:17.456851 6540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:17.466528 6540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:17.467809 6540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:17.469074 6540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:17.482281 6545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:17.482372 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:17.482350 6546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:17.483168 6540 server_base.cc:1034] running on GCE node
I20260808 15:45:17.484062 6540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:17.485539 6540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:17.486747 6540 hybrid_clock.cc:648] HybridClock initialized: now 1786203917486708 us; error 32 us; skew 500 ppm
I20260808 15:45:17.487135 6540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:17 dist-test-slave-gp6n krb5kdc[6526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203917, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:17.495553 6540 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:17.497217 6540 webserver.cc:460] Webserver started at http://127.23.130.62:39225/ using document root <none> and password file <none>
I20260808 15:45:17.497956 6540 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:17.498107 6540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:17.498488 6540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:17.501608 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "63d0a2af1e934d62b30293f1fda609d0"
format_stamp: "Formatted at 2026-08-08 15:45:17 on dist-test-slave-gp6n"
server_key: "3ac01bde76c11530c60e0d0b1a26d959"
server_key_iv: "676119b7ecc834f2f17c34a279cf084d"
server_key_version: "encryptionkey@0"
I20260808 15:45:17.502547 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "63d0a2af1e934d62b30293f1fda609d0"
format_stamp: "Formatted at 2026-08-08 15:45:17 on dist-test-slave-gp6n"
server_key: "3ac01bde76c11530c60e0d0b1a26d959"
server_key_iv: "676119b7ecc834f2f17c34a279cf084d"
server_key_version: "encryptionkey@0"
I20260808 15:45:17.507797 6540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:45:17.511381 6555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:17.512826 6540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:45:17.513020 6540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "63d0a2af1e934d62b30293f1fda609d0"
format_stamp: "Formatted at 2026-08-08 15:45:17 on dist-test-slave-gp6n"
server_key: "3ac01bde76c11530c60e0d0b1a26d959"
server_key_iv: "676119b7ecc834f2f17c34a279cf084d"
server_key_version: "encryptionkey@0"
I20260808 15:45:17.513223 6540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:17.541990 6540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:17.546294 6540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:17.546631 6540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:17.566118 6540 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:38255
I20260808 15:45:17.566157 6610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:38255 every 8 connection(s)
I20260808 15:45:17.567492 6540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:17.571215 6611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:17.573282 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6540
I20260808 15:45:17.573551 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:45:17.574391 24072 external_mini_cluster.cc:1428] Setting key 10ea31f45ceb3f1aec242721300cf373
I20260808 15:45:17.580804 6611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:45:17 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203917, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.61@KRBTEST.COM" created.
I20260808 15:45:17.606366 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.584825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38255 (local address 127.0.0.1:53534)
0808 15:45:17.585783 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.585839 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:17.588204 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:17.589153 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:17.589158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:17.589731 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:17.590074 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:17.590091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:17.592503 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:17.592506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:17.593598 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:17.593610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:17.595001 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:17.595006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:17.595118 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:17.595808 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:17.595830 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:17.597989 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:17.602479 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:17.602490 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:17.602499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:17.602896 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:17.603417 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:17.603421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:17.603423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:17.603588 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:17.604419 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:17.604425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:17.604611 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 15:45:17.604820 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 15:45:17.605478 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":155,"thread_start_us":402,"threads_started":1}
I20260808 15:45:17.606449 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.584825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.23.130.62:38255)
0808 15:45:17.586779 (+ 1954us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:17.586785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:17.586826 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:45:17.588577 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:17.588588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:17.588884 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:17.589113 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:17.590283 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:17.592345 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:17.593748 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:17.594769 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:17.594971 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:17.598239 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:17.598252 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:17.598271 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:17.598317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:17.602282 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:17.603045 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:17.603056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:17.603061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:17.603263 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:17.603694 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:17.603698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:17.603700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:17.604258 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:17.604419 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:17.604960 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 15:45:17.605228 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1755,"spinlock_wait_cycles":15616,"thread_start_us":1524,"threads_started":1}
I20260808 15:45:17.609479 6611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:45:17.610746 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.610236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47414)
0808 15:45:17.610460 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.610595 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 15:45:17.611056 6559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
I20260808 15:45:17.612802 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.612344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47420)
0808 15:45:17.612524 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.612617 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 15:45:17.613009 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39273: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
I20260808 15:45:17.637964 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:39273
--webserver_interface=127.23.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:17.662499 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } attempt: 1
I20260808 15:45:17.663662 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.663180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47436)
0808 15:45:17.663384 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.663529 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:45:17.664731 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.664358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47440)
0808 15:45:17.664498 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.664577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:45:17.664907 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39273: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
I20260808 15:45:17.733189 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } attempt: 2
I20260808 15:45:17.734498 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.733995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47450)
0808 15:45:17.734195 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.734287 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 15:45:17.735527 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.735154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47454)
0808 15:45:17.735301 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.735390 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:45:17.735716 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39273: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
I20260808 15:45:17.826978 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } attempt: 3
I20260808 15:45:17.828182 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.827725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47470)
0808 15:45:17.827937 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.828048 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 15:45:17.829252 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.828895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47476)
0808 15:45:17.829047 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.829114 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:45:17.829514 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39273: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
I20260808 15:45:17.972738 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } attempt: 4
I20260808 15:45:17.973985 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.973392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47486)
0808 15:45:17.973747 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.973835 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 15:45:17.975049 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:17.974685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47496)
0808 15:45:17.974833 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:17.974926 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:45:17.975231 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39273: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39273: connect: Connection refused (error 111)
W20260808 15:45:17.975627 6619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:17.975946 6619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:17.976099 6619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:17.983332 6619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:17.983482 6619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:17.983585 6619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:17.983688 6619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:17.983777 6619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:17.993521 6619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:17.994900 6619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:17.996217 6619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:18.007988 6624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:18.008163 6627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:18.008204 6625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:18.009377 6619 server_base.cc:1034] running on GCE node
I20260808 15:45:18.010125 6619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:18.011588 6619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:18.012801 6619 hybrid_clock.cc:648] HybridClock initialized: now 1786203918012763 us; error 57 us; skew 500 ppm
I20260808 15:45:18.013150 6619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:18.016273 6619 init.cc:377] Logged in from keytab as kudu/127.23.130.61@KRBTEST.COM (short username kudu)
I20260808 15:45:18.017813 6619 webserver.cc:460] Webserver started at http://127.23.130.61:45911/ using document root <none> and password file <none>
I20260808 15:45:18.018517 6619 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:18.018622 6619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:18.018872 6619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:18.021732 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data/instance:
uuid: "70ec95ab56724cb6a7f6e7af9c837fce"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "ec7cb1e743e6ea9c68faf87ad4b3116e"
server_key_iv: "1eee6eff4c8b5c84dab30392e0a3ce4c"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.022624 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal/instance:
uuid: "70ec95ab56724cb6a7f6e7af9c837fce"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "ec7cb1e743e6ea9c68faf87ad4b3116e"
server_key_iv: "1eee6eff4c8b5c84dab30392e0a3ce4c"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.027608 6619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:45:18.031450 6634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:18.032943 6619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:45:18.033145 6619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "70ec95ab56724cb6a7f6e7af9c837fce"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "ec7cb1e743e6ea9c68faf87ad4b3116e"
server_key_iv: "1eee6eff4c8b5c84dab30392e0a3ce4c"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.033373 6619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:18.046991 6619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:18.050984 6619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:18.051273 6619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:18.070694 6619 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.61:39273
I20260808 15:45:18.070724 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.61:39273 every 8 connection(s)
I20260808 15:45:18.072084 6619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
I20260808 15:45:18.072957 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6619
I20260808 15:45:18.073199 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-1/wal/instance
I20260808 15:45:18.073644 24072 external_mini_cluster.cc:1428] Setting key c6569bcd69ccc0b642d0d250fe993b44
I20260808 15:45:18.075973 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:18.082584 6690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.60@KRBTEST.COM" created.
I20260808 15:45:18.102927 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.085129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.23.130.62:38255)
0808 15:45:18.085307 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.085311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.086699 (+ 1388us) server_negotiation.cc:407] Connection header received
0808 15:45:18.088645 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.088651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.088747 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.088863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.090378 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.091133 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.092685 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.093116 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.093256 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.096782 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.096787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.096791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.096823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.099763 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.100576 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.100582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.100585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.100754 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.101182 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.101188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.101191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.101484 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.101656 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.102331 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.102604 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:45:18.103914 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.085714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38255 (local address 127.0.0.1:53536)
0808 15:45:18.086534 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.086574 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.088344 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.089097 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.089105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.089637 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.090195 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.090213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.091307 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.091311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.092533 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.092546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.093366 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.093374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.093607 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.094317 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.094340 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.096548 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.099903 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.099911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.099928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.100432 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.100865 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.100870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.100874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.101065 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.101617 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.101625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.101973 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.102243 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.102763 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":323,"threads_started":1}
I20260808 15:45:18.105557 6690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:45:18.120424 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.106389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47512)
0808 15:45:18.106584 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.106606 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.106742 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.107576 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.107580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.107926 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.108066 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.108073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.110022 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.110025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.110948 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.110960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.111769 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.111774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.111979 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.112550 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.112566 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.114514 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.117847 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.117852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.117857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.118297 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.118692 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.118698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.118702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.118850 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.119448 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.119452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.119603 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.119749 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.120020 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 15:45:18.120519 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.106464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.23.130.61:39273)
0808 15:45:18.107078 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.107088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.107122 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:45:18.107178 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.107188 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.107420 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.107574 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.108219 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.109888 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.111088 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.111605 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.111754 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.114708 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.114720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.114731 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.114762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.117656 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.118452 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.118467 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.118470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.118581 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.118984 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.118988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.118989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.119324 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.119540 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.119912 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.120128 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":254,"threads_started":1}
I20260808 15:45:18.123106 6690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:45:18.124397 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.123927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42722)
0808 15:45:18.124137 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.124248 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 15:45:18.124711 6637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.126307 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.125978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42738)
0808 15:45:18.126123 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.126203 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:45:18.126555 6690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
I20260808 15:45:18.134270 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:42569
--webserver_interface=127.23.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:18.175849 6690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 1
I20260808 15:45:18.177119 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.176645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42746)
0808 15:45:18.176847 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.176958 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 15:45:18.178144 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.177781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42756)
0808 15:45:18.177957 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.178030 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:45:18.178359 6690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.246609 6690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 2
I20260808 15:45:18.247756 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.247268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42768)
0808 15:45:18.247496 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.247591 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 15:45:18.248766 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.248418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42784)
0808 15:45:18.248574 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.248649 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 15:45:18.248975 6690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.274569 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } attempt: 5
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203917, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:45:18.287930 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.275450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.23.130.61:39273)
0808 15:45:18.275667 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.275675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.275770 (+ 95us) server_negotiation.cc:407] Connection header received
0808 15:45:18.275987 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.275994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.276080 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.276189 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.276997 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.277696 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.278935 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.279397 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.279547 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.282655 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.282660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.282663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.282694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.285326 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.285972 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.285980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.285986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.286114 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.286519 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.286525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.286528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.286807 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.286969 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.287389 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.287615 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 15:45:18.287931 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.275430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47522)
0808 15:45:18.275653 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.275676 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.275852 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.276228 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.276234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.276635 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.276827 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.276835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.277834 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.277841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.278791 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.278804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.279559 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.279564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.279754 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.280402 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.280419 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.282458 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.285482 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.285487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.285491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.285855 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.286229 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.286234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.286238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.286380 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.286933 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.286938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.287113 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.287270 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.287557 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 15:45:18.289316 6611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:45:18.290442 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.290040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42790)
0808 15:45:18.290198 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.290271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:45:18.291332 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.291010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42806)
0808 15:45:18.291159 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.291234 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 15:45:18.291533 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.340179 6690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 3
I20260808 15:45:18.341370 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.340900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42820)
0808 15:45:18.341111 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.341210 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 15:45:18.342505 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.342014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42826)
0808 15:45:18.342246 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.342324 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260808 15:45:18.342711 6690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.342752 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 1
I20260808 15:45:18.343722 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.343314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42842)
0808 15:45:18.343483 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.343580 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:45:18.344679 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.344349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42850)
0808 15:45:18.344493 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.344575 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:45:18.344960 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.413213 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 2
I20260808 15:45:18.414590 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.413935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42858)
0808 15:45:18.414205 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.414327 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 15:45:18.415691 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.415271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42872)
0808 15:45:18.415441 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.415534 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:45:18.415917 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
I20260808 15:45:18.485926 6690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 4
I20260808 15:45:18.487149 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.486684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42876)
0808 15:45:18.486904 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.486997 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 15:45:18.488212 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.487909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42886)
0808 15:45:18.488039 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.488110 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 15:45:18.488454 6690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
W20260808 15:45:18.493848 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:18.494158 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:18.494325 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:18.501770 6698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:18.501868 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:18.501974 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:18.502075 6698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:18.502157 6698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:18.512418 6698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38255,127.23.130.61:39273,127.23.130.60:42569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:18.513708 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:18.515106 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:45:18.522140 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 3
I20260808 15:45:18.523234 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.522774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42892)
0808 15:45:18.522995 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.523088 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:45:18.524295 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.523887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42904)
0808 15:45:18.524096 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.524176 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260808 15:45:18.524501 6611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:42569: Network error: Client connection negotiation failed: client connection to 127.23.130.60:42569: connect: Connection refused (error 111)
W20260808 15:45:18.527278 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:18.527495 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:18.527310 6708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:18.528429 6698 server_base.cc:1034] running on GCE node
I20260808 15:45:18.529210 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:18.530680 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:18.531949 6698 hybrid_clock.cc:648] HybridClock initialized: now 1786203918531900 us; error 39 us; skew 500 ppm
I20260808 15:45:18.532305 6698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:18.535445 6698 init.cc:377] Logged in from keytab as kudu/127.23.130.60@KRBTEST.COM (short username kudu)
I20260808 15:45:18.536818 6698 webserver.cc:460] Webserver started at http://127.23.130.60:46707/ using document root <none> and password file <none>
I20260808 15:45:18.537443 6698 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:18.537606 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:18.537950 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:18.540874 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data/instance:
uuid: "277a93cb000743ecb46ed50574c82889"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "f982afd9e27b2f966aef1af370fe880e"
server_key_iv: "8694507928d15b0145f4401225772a56"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.541683 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal/instance:
uuid: "277a93cb000743ecb46ed50574c82889"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "f982afd9e27b2f966aef1af370fe880e"
server_key_iv: "8694507928d15b0145f4401225772a56"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.546838 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:45:18.550364 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:18.551901 6698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:45:18.552067 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "277a93cb000743ecb46ed50574c82889"
format_stamp: "Formatted at 2026-08-08 15:45:18 on dist-test-slave-gp6n"
server_key: "f982afd9e27b2f966aef1af370fe880e"
server_key_iv: "8694507928d15b0145f4401225772a56"
server_key_version: "encryptionkey@0"
I20260808 15:45:18.552284 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:18.574167 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:18.578536 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:18.578824 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:18.596666 6698 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.60:42569
I20260808 15:45:18.596704 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.60:42569 every 8 connection(s)
I20260808 15:45:18.598061 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
I20260808 15:45:18.601946 6772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:18.602241 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6698
I20260808 15:45:18.602478 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786203706771851-24072-0/minicluster-data/master-2/wal/instance
I20260808 15:45:18.603012 24072 external_mini_cluster.cc:1428] Setting key d3a885f3c85105bc40c530d95ad4a224
I20260808 15:45:18.612074 6772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:18.628194 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.605084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.23.130.60:42569)
0808 15:45:18.605944 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.605972 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.606027 (+ 55us) server_negotiation.cc:407] Connection header received
0808 15:45:18.608854 (+ 2827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.608883 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.609421 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.609885 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.611364 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.613408 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.615074 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.616221 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.616411 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.621008 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.621016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.621028 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.621063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.624105 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.624858 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.624864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.624870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.625009 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.625430 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.625435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.625437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.625943 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.626250 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.626757 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.627172 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":357,"threads_started":1}
I20260808 15:45:18.630400 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.614949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38255 (local address 127.0.0.1:53552)
0808 15:45:18.615942 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.615984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.616176 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.616625 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.616633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.617481 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.617836 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.617847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.619047 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.619052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.620408 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.620421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.621178 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.621186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.621378 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.622122 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.622138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.624490 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.627803 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.627815 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.627823 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.628289 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.628712 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.628717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.628726 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.628860 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.629456 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.629463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.629678 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.629851 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.630078 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":634,"threads_started":1}
I20260808 15:45:18.630702 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.614957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.23.130.62:38255)
0808 15:45:18.615615 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.615623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.616084 (+ 461us) server_negotiation.cc:407] Connection header received
0808 15:45:18.616321 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.616325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.616391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.616504 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.618005 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.618892 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.620558 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.620980 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.621119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.624690 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.624697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.624701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.624741 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.627675 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.628464 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.628468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.628470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.628592 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.629012 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.629021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.629023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.629292 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.629495 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.629999 (+ 504us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.630225 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":356,"threads_started":1}
I20260808 15:45:18.632115 6772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:18.645577 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.632878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39273 (local address 127.0.0.1:47534)
0808 15:45:18.633039 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.633056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.633142 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.633467 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.633471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.633781 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.633913 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.633920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.634999 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.635004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.635870 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.635882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.636599 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.636605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.636787 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.637372 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.637387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.639878 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.643170 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.643175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.643179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.643726 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.644112 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.644116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.644119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.644296 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.644802 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.644806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.644995 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.645133 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.645340 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 15:45:18.645907 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.632902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.23.130.61:39273)
0808 15:45:18.633105 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.633111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.633133 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:45:18.633234 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.633242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.633328 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.633435 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.634098 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.634849 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.636019 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.636432 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.636571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.640082 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.640089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.640095 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.640135 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.643029 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.643846 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.643850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.643852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.643992 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.644417 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.644420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.644422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.644682 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.644792 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.645270 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.645507 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:45:18.646955 6772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:45:18.648208 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.631854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.23.130.62:38255)
0808 15:45:18.632003 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.632009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.632081 (+ 72us) server_negotiation.cc:407] Connection header received
0808 15:45:18.632250 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.632255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.632339 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.632449 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.634036 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.634808 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.636096 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.636506 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.636654 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.641600 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.641604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.641607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.641678 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.644860 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.645740 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.645746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.645750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.645897 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.646355 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.646360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.646363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.646778 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.647051 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.647617 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.647872 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:45:18.661052 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.647810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.23.130.60:42569)
0808 15:45:18.647992 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.647998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.648156 (+ 158us) server_negotiation.cc:407] Connection header received
0808 15:45:18.648331 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.648338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.648412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.648491 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.649415 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.650343 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.651584 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.652001 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.652139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.655083 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.655091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.655096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.655132 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.658173 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.659027 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.659036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.659041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.659171 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.659560 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.659568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.659572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.659877 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.660039 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.660387 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.660632 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:45:18.661238 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.647775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42930)
0808 15:45:18.648061 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.648080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.648203 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.648672 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.648675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.649046 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.649246 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.649258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.650519 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.650525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.651447 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.651459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.652153 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.652159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.652349 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.652894 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.652910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.654874 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.658351 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.658355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.658357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.658904 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.659295 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.659310 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.659316 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.659448 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.659998 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.660003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.660147 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.660318 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.660804 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":158}
I20260808 15:45:18.665684 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.649952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.23.130.61:39273)
0808 15:45:18.650134 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.650141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.650160 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:45:18.650324 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.650330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.650451 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.650550 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.651928 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.652699 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.654075 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.654659 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.654807 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.659805 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.659810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.659813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.659844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.662906 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.663682 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.663688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.663691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.663853 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.664223 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.664228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.664232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.664567 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.664836 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.665104 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.665360 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:45:18.667582 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: Bootstrap starting.
I20260808 15:45:18.671020 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:45:18.671600 6772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:18.672616 6772 log.cc:824] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:18.675534 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: No bootstrap required, opened a new log
I20260808 15:45:18.690035 6772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.690606 6772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:18.690765 6772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277a93cb000743ecb46ed50574c82889, State: Initialized, Role: FOLLOWER
I20260808 15:45:18.691493 6772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.700201 6772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:18.701715 6611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 4
I20260808 15:45:18.703815 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.676715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.23.130.62:38255)
0808 15:45:18.676973 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.676983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.683270 (+ 6287us) server_negotiation.cc:407] Connection header received
0808 15:45:18.683753 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.683759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.683845 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.683922 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.686520 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.687621 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.690899 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.691565 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.691668 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.696043 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.696052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.696060 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.696102 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.700110 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.701083 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.701091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.701098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.701251 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.701818 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.701824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.701830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.702264 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.702562 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.703183 (+ 621us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.703428 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:45:18.708930 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.678874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.23.130.60:42569)
0808 15:45:18.679084 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.679090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.687355 (+ 8265us) server_negotiation.cc:407] Connection header received
0808 15:45:18.687554 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.687559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.687646 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.687748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.690394 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.691598 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.695045 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.696120 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.696252 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.700390 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.700407 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.700414 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.700455 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.704170 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.705389 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.705399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.705406 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.705667 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.706265 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.706273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.706280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.706838 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.707184 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.707989 (+ 805us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.708318 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:45:18.709303 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.676228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.23.130.61:39273)
0808 15:45:18.676464 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.676473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.686890 (+ 10417us) server_negotiation.cc:407] Connection header received
0808 15:45:18.687165 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.687171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.687244 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.687307 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.689636 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.690856 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.692858 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.693543 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.693651 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.697322 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.697331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.697339 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.697392 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.701043 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.702217 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.702223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.702226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.704832 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.706292 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.706297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.706301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.706835 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.707135 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.708466 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.708779 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:45:18.711496 6789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:18.711930 6789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203917, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
W20260808 15:45:18.730634 6803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:45:18.730849 6803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:45:18.731380 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.704506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.23.130.60:42569)
0808 15:45:18.709675 (+ 5169us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.709683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.709706 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:45:18.709771 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.709779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.709872 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.711371 (+ 1499us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.711582 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.713280 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.715734 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.716497 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.716642 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.722667 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.722673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.722679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.722749 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.727497 (+ 4748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.728487 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.728495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.728501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.728669 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.729187 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.729192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.729196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.729787 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.729974 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.730632 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.730900 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5026,"thread_start_us":11462,"threads_started":1}
I20260808 15:45:18.732733 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.702748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42938)
0808 15:45:18.703203 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.703228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.703373 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.710100 (+ 6727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.710110 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.710860 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.711152 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.711169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.713472 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.713481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.715511 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.715532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.716652 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.716658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.716841 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.717529 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.717563 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.722343 (+ 4780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.727660 (+ 5317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.727669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.727677 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.728312 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.728823 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.728830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.728837 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.729041 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.730067 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.730073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.730254 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.731897 (+ 1643us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.732223 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260808 15:45:18.738965 6611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: Bootstrap starting.
I20260808 15:45:18.727049 6698 master_runner.cc:186] Error getting master registration for 127.23.130.62:38255: OK, instance_id { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" instance_seqno: 1786203917560695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:45:18.740278 6698 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:45:18.743501 6611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:18.744544 6611 log.cc:824] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:18.747366 6611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: No bootstrap required, opened a new log
I20260808 15:45:18.755929 6611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.756451 6611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:18.756595 6611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d0a2af1e934d62b30293f1fda609d0, State: Initialized, Role: FOLLOWER
I20260808 15:45:18.757256 6611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.758214 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:18.758658 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:18.759495 6611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:45:18.771638 6817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:45:18.771859 6817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:45:18.771274 6540 master_runner.cc:186] Error getting master registration for 127.23.130.61:39273: OK, instance_id { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" instance_seqno: 1786203918064951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:45:18.785211 6540 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:45:18.787734 6690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } attempt: 5
Aug 08 15:45:18 dist-test-slave-gp6n krb5kdc[6526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203918, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
W20260808 15:45:18.799638 6561 tablet.cc:2369] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:18.808087 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.788293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:42569 (local address 127.0.0.1:42946)
0808 15:45:18.788852 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:18.788873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:18.789002 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:18.789393 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:18.789400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:18.789960 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:18.790181 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.790194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.791833 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.791839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.793256 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:18.793272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.794382 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.794388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:18.794673 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.795413 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:18.795438 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:18.798455 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:18.804858 (+ 6403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.804864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.804869 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.805453 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.805903 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:18.805911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:18.805916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:18.806096 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:18.806941 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:18.806947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:18.807136 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 15:45:18.807331 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 15:45:18.807652 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412}
I20260808 15:45:18.808102 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:18.788641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.23.130.60:42569)
0808 15:45:18.788857 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:18.788864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:18.788958 (+ 94us) server_negotiation.cc:407] Connection header received
0808 15:45:18.789135 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:18.789141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:18.789221 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:18.789366 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:18.790358 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.791660 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.793442 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:18.794204 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:18.794365 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:18.798650 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:18.798657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:18.798663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:18.798705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:18.804718 (+ 6013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.805588 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.805594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.805598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.805772 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:18.806314 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:18.806319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:18.806323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:18.806793 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:18.806989 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:18.807440 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:45:18.807676 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:45:18.814997 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: Bootstrap starting.
I20260808 15:45:18.819940 6690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:18.821381 6690 log.cc:824] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:18.824927 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: No bootstrap required, opened a new log
W20260808 15:45:18.830796 6722 tablet.cc:2369] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:18.837881 6690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.838488 6690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:18.838634 6690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ec95ab56724cb6a7f6e7af9c837fce, State: Initialized, Role: FOLLOWER
I20260808 15:45:18.839432 6690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.842273 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:18.842780 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:18.843703 6690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:18.864866 6619 master_runner.cc:263] No leader master found from master 70ec95ab56724cb6a7f6e7af9c837fce
I20260808 15:45:18.865028 6619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 15:45:18.869102 6833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:45:18.869242 6833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:45:18.878996 6821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:45:18.879241 6821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.880760 6821 leader_election.cc:288] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63d0a2af1e934d62b30293f1fda609d0 (127.23.130.62:38255), 277a93cb000743ecb46ed50574c82889 (127.23.130.60:42569)
I20260808 15:45:18.882068 6583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d0a2af1e934d62b30293f1fda609d0" is_pre_election: true
I20260808 15:45:18.882773 6746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277a93cb000743ecb46ed50574c82889" is_pre_election: true
I20260808 15:45:18.883388 6746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ec95ab56724cb6a7f6e7af9c837fce in term 0.
I20260808 15:45:18.884150 6637 leader_election.cc:302] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277a93cb000743ecb46ed50574c82889, 70ec95ab56724cb6a7f6e7af9c837fce; no voters:
I20260808 15:45:18.884820 6583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ec95ab56724cb6a7f6e7af9c837fce in term 0.
I20260808 15:45:18.885665 6821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:45:18.885819 6821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:45:18.885927 6821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:18.889533 6821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.891089 6583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d0a2af1e934d62b30293f1fda609d0"
I20260808 15:45:18.891307 6583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:18.894760 6583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ec95ab56724cb6a7f6e7af9c837fce in term 1.
I20260808 15:45:18.895265 6636 leader_election.cc:302] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d0a2af1e934d62b30293f1fda609d0, 70ec95ab56724cb6a7f6e7af9c837fce; no voters:
I20260808 15:45:18.901640 6746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277a93cb000743ecb46ed50574c82889"
I20260808 15:45:18.901880 6746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:18.905357 6746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ec95ab56724cb6a7f6e7af9c837fce in term 1.
I20260808 15:45:18.906186 6821 leader_election.cc:288] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [CANDIDATE]: Term 1 election: Requested vote from peers 63d0a2af1e934d62b30293f1fda609d0 (127.23.130.62:38255), 277a93cb000743ecb46ed50574c82889 (127.23.130.60:42569)
I20260808 15:45:18.906579 6821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:18.907004 6821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [term 1 LEADER]: Becoming Leader. State: Replica: 70ec95ab56724cb6a7f6e7af9c837fce, State: Running, Role: LEADER
I20260808 15:45:18.907832 6821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } }
I20260808 15:45:18.923952 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ec95ab56724cb6a7f6e7af9c837fce. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:18.924492 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:18.934952 6837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:18.936568 6837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:18.964581 6746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [term 1 FOLLOWER]: Refusing update from remote peer 70ec95ab56724cb6a7f6e7af9c837fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:18.967079 6835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:18.974265 6583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [term 1 FOLLOWER]: Refusing update from remote peer 70ec95ab56724cb6a7f6e7af9c837fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:18.975288 6835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:18.988967 6789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ec95ab56724cb6a7f6e7af9c837fce. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:18.989317 6789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:18.993896 6838 mvcc.cc:205] Tried to move back new op lower bound from 7316291252031143936 to 7316291251859714048. Current Snapshot: MvccSnapshot[applied={T|T < 7316291252031143936}]
I20260808 15:45:19.010613 6789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:19.010902 6789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:19.010838 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:19.011119 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:19.011482 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:19.011698 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:19.012750 6837 catalog_manager.cc:1347] Generated new cluster ID: 3f9343ff46a344e4bb7f0c5dcbc03f68
I20260808 15:45:19.012854 6837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:19.031375 6845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:19.031824 6845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:19.033423 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ec95ab56724cb6a7f6e7af9c837fce. Latest consensus state: current_term: 1 leader_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d0a2af1e934d62b30293f1fda609d0" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38255 } } peers { permanent_uuid: "70ec95ab56724cb6a7f6e7af9c837fce" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39273 } } peers { permanent_uuid: "277a93cb000743ecb46ed50574c82889" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 42569 } } }
I20260808 15:45:19.033705 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:45:19.049404 6837 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:45:19.051563 6837 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:19.074702 6837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ec95ab56724cb6a7f6e7af9c837fce: Generated new TSK 0
I20260808 15:45:19.075764 6837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:45:19.733012 6803 catalog_manager.cc:1259] Loaded cluster ID: 3f9343ff46a344e4bb7f0c5dcbc03f68
I20260808 15:45:19.733187 6803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: loading cluster ID for follower catalog manager: success
I20260808 15:45:19.735669 6803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: acquiring CA information for follower catalog manager: success
I20260808 15:45:19.737587 6803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 277a93cb000743ecb46ed50574c82889: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:45:19.774012 6817 catalog_manager.cc:1259] Loaded cluster ID: 3f9343ff46a344e4bb7f0c5dcbc03f68
I20260808 15:45:19.774164 6817 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: loading cluster ID for follower catalog manager: success
I20260808 15:45:19.776692 6817 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: acquiring CA information for follower catalog manager: success
I20260808 15:45:19.778551 6817 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63d0a2af1e934d62b30293f1fda609d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:45:20.221632 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6540
I20260808 15:45:20.223289 6605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:20.334472 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6540
I20260808 15:45:20.359802 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6619
I20260808 15:45:20.361011 6684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:20.471920 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6619
W20260808 15:45:20.496187 6717 connection.cc:535] server connection from 127.0.0.1:42946 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:45:20.496542 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6698
I20260808 15:45:20.497709 6767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:20.607542 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6698
2026-08-08T15:45:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4992 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:45:20 dist-test-slave-gp6n krb5kdc[6857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:45:20 dist-test-slave-gp6n krb5kdc[6857](info): set up 2 sockets
Aug 08 15:45:20 dist-test-slave-gp6n krb5kdc[6857](info): commencing operation
krb5kdc: starting...
W20260808 15:45:21.776716 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:45:21 dist-test-slave-gp6n krb5kdc[6857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:45:22.058377 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42285
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:22.400064 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:22.400388 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:22.400482 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:22.408038 6870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:22.408159 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:22.408213 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:22.408272 6870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:22.408317 6870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:22.418277 6870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:42285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:22.419579 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:22.420863 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:22.432729 6877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:22.432807 6878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:22.432811 6880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:22.433812 6870 server_base.cc:1034] running on GCE node
I20260808 15:45:22.434809 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:22.436254 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:22.437500 6870 hybrid_clock.cc:648] HybridClock initialized: now 1786203922437455 us; error 56 us; skew 500 ppm
I20260808 15:45:22.437851 6870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:22 dist-test-slave-gp6n krb5kdc[6857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203922, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:22.441030 6870 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:22.442883 6870 webserver.cc:460] Webserver started at http://127.23.130.62:34607/ using document root <none> and password file <none>
I20260808 15:45:22.443579 6870 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:22.443732 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:22.444057 6870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:22.447221 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "cfb987d3a79c4cee808f43e0407a6863"
format_stamp: "Formatted at 2026-08-08 15:45:22 on dist-test-slave-gp6n"
server_key: "7a9f8e68de684013cb37eead7d6c1f76"
server_key_iv: "47d58fea38a65232cf41c9f65fc90f58"
server_key_version: "encryptionkey@0"
I20260808 15:45:22.448006 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "cfb987d3a79c4cee808f43e0407a6863"
format_stamp: "Formatted at 2026-08-08 15:45:22 on dist-test-slave-gp6n"
server_key: "7a9f8e68de684013cb37eead7d6c1f76"
server_key_iv: "47d58fea38a65232cf41c9f65fc90f58"
server_key_version: "encryptionkey@0"
I20260808 15:45:22.453203 6870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:45:22.456791 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:22.458211 6870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:45:22.458431 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "cfb987d3a79c4cee808f43e0407a6863"
format_stamp: "Formatted at 2026-08-08 15:45:22 on dist-test-slave-gp6n"
server_key: "7a9f8e68de684013cb37eead7d6c1f76"
server_key_iv: "47d58fea38a65232cf41c9f65fc90f58"
server_key_version: "encryptionkey@0"
I20260808 15:45:22.458652 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:22.476763 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:22.480887 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:22.481206 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:22.500842 6870 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:42285
I20260808 15:45:22.500936 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:42285 every 8 connection(s)
I20260808 15:45:22.502251 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:22.505743 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6870
I20260808 15:45:22.505988 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:45:22.506523 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:22.506814 24072 external_mini_cluster.cc:1428] Setting key 50b5a442f4426a39e11dc4875746355c
I20260808 15:45:22.521641 6946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863: Bootstrap starting.
I20260808 15:45:22.525520 6946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:22.526970 6946 log.cc:824] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:45:22 dist-test-slave-gp6n krb5kdc[6857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:22.530043 6946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863: No bootstrap required, opened a new log
I20260808 15:45:22.535977 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:22.508618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.23.130.62:42285)
0808 15:45:22.509400 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:22.509412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:22.509454 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:45:22.511434 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:22.511455 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:22.511837 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:22.512146 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:22.514221 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:22.515626 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:22.517709 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:22.518498 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:22.518645 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:22.528376 (+ 9731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:22.528385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:22.528399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:22.528441 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:22.531611 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:22.532416 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:22.532423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:22.532428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:22.532615 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:22.533034 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:22.533039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:22.533052 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:22.533640 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:22.533966 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:22.534497 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 15:45:22.534941 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":352,"threads_started":1}
I20260808 15:45:22.539073 6946 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } }
I20260808 15:45:22.539498 6946 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:22.539628 6946 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb987d3a79c4cee808f43e0407a6863, State: Initialized, Role: FOLLOWER
I20260808 15:45:22.539865 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:45:22.540280 6946 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } }
I20260808 15:45:22.540558 6946 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:22.540707 6946 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:22.540899 6946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:22.543437 6946 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } }
I20260808 15:45:22.543957 6946 leader_election.cc:302] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfb987d3a79c4cee808f43e0407a6863; no voters:
I20260808 15:45:22.544552 6946 leader_election.cc:288] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:45:22.544678 6952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:22.545416 6952 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [term 1 LEADER]: Becoming Leader. State: Replica: cfb987d3a79c4cee808f43e0407a6863, State: Running, Role: LEADER
I20260808 15:45:22.545994 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } }
I20260808 15:45:22.546856 6946 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:22.548815 6954 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfb987d3a79c4cee808f43e0407a6863. Latest consensus state: current_term: 1 leader_uuid: "cfb987d3a79c4cee808f43e0407a6863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } } }
I20260808 15:45:22.549048 6953 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfb987d3a79c4cee808f43e0407a6863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb987d3a79c4cee808f43e0407a6863" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42285 } } }
I20260808 15:45:22.549329 6954 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:22.549329 6953 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:22.550956 6958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:22.555905 6958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:22.564951 6958 catalog_manager.cc:1347] Generated new cluster ID: 63253a0698e049d2811c2a80c4f3592a
I20260808 15:45:22.565073 6958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:22.579385 6958 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:45:22.580667 6958 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:22.596226 6958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfb987d3a79c4cee808f43e0407a6863: Generated new TSK 0
I20260808 15:45:22.597023 6958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:45:22.652803 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6870
I20260808 15:45:22.654106 6941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:22.751699 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6870
2026-08-08T15:45:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2143 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:45:22 dist-test-slave-gp6n krb5kdc[6977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:45:22 dist-test-slave-gp6n krb5kdc[6977](info): set up 2 sockets
Aug 08 15:45:22 dist-test-slave-gp6n krb5kdc[6977](info): commencing operation
krb5kdc: starting...
W20260808 15:45:23.981624 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:45:24 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:45:24.262674 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:41991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:24.600091 6993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:24.600369 6993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:24.600467 6993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:24.607473 6993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:24.607578 6993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:24.607625 6993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:24.607683 6993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:24.607720 6993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:24.617112 6993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:41991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:24.618443 6993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:24.619664 6993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:24.631559 6998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:24.631610 7001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:24.631507 6999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:24.633220 6993 server_base.cc:1034] running on GCE node
I20260808 15:45:24.634158 6993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:24.635856 6993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:24.637091 6993 hybrid_clock.cc:648] HybridClock initialized: now 1786203924637020 us; error 56 us; skew 500 ppm
I20260808 15:45:24.637477 6993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:24 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:24.641002 6993 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:24.642645 6993 webserver.cc:460] Webserver started at http://127.23.130.62:40319/ using document root <none> and password file <none>
I20260808 15:45:24.643299 6993 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:24.643411 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:24.643692 6993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:24.646924 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "b8c5a7613d44459198cd4935c67984a4"
format_stamp: "Formatted at 2026-08-08 15:45:24 on dist-test-slave-gp6n"
server_key: "16c453475c8e9e499ba96c013c15cd59"
server_key_iv: "c1d6b95fd59858a9f4dd47c6a668cffb"
server_key_version: "encryptionkey@0"
I20260808 15:45:24.647650 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "b8c5a7613d44459198cd4935c67984a4"
format_stamp: "Formatted at 2026-08-08 15:45:24 on dist-test-slave-gp6n"
server_key: "16c453475c8e9e499ba96c013c15cd59"
server_key_iv: "c1d6b95fd59858a9f4dd47c6a668cffb"
server_key_version: "encryptionkey@0"
I20260808 15:45:24.653019 6993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:45:24.656436 7008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:24.657783 6993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:45:24.657943 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "b8c5a7613d44459198cd4935c67984a4"
format_stamp: "Formatted at 2026-08-08 15:45:24 on dist-test-slave-gp6n"
server_key: "16c453475c8e9e499ba96c013c15cd59"
server_key_iv: "c1d6b95fd59858a9f4dd47c6a668cffb"
server_key_version: "encryptionkey@0"
I20260808 15:45:24.658154 6993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:24.696439 6993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:24.700596 6993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:24.700831 6993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:24.720054 6993 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:41991
I20260808 15:45:24.720103 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:41991 every 8 connection(s)
I20260808 15:45:24.721470 6993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:24.725306 7062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:24.729504 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 6993
I20260808 15:45:24.729740 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:45:24.730463 24072 external_mini_cluster.cc:1428] Setting key 3cee796d76a4b463b183462b163fe773
I20260808 15:45:24.736552 7062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap starting.
I20260808 15:45:24.740345 7062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:24.741559 7062 log.cc:824] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:24.744467 7062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: No bootstrap required, opened a new log
Aug 08 15:45:24 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:24.753754 7062 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:24.754141 7062 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:24.754261 7062 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Initialized, Role: FOLLOWER
I20260808 15:45:24.754848 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:24.732494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.23.130.62:41991)
0808 15:45:24.733312 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:24.733324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:24.733369 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:45:24.735201 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:24.735224 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:24.735597 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:24.735908 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:24.737842 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:24.739271 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:24.741201 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:24.741985 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:24.742232 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:24.746976 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:24.746985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:24.747002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:24.747048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:24.750362 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:24.751250 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:24.751256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:24.751262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:24.751451 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:24.751930 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:24.751933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:24.751935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:24.752530 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:24.752790 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:24.753235 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 15:45:24.753619 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":390,"threads_started":1}
I20260808 15:45:24.754976 7062 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:24.755254 7062 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:24.755431 7062 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:24.755628 7062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:24.758738 7062 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:24.759263 7062 leader_election.cc:302] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c5a7613d44459198cd4935c67984a4; no voters:
I20260808 15:45:24.759891 7062 leader_election.cc:288] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:45:24.760035 7067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:24.761322 7067 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 LEADER]: Becoming Leader. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Running, Role: LEADER
I20260808 15:45:24.761931 7067 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:24.762127 7062 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:24.765056 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:24.765714 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:24.765161 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c5a7613d44459198cd4935c67984a4. Latest consensus state: current_term: 1 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:24.765991 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:24.768594 7076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:24.774166 7076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:24.783708 7076 catalog_manager.cc:1347] Generated new cluster ID: 93d36f4984694822b871be7dd019848a
I20260808 15:45:24.783833 7076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:24.801777 7076 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:45:24.803725 7076 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:24.819278 7076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Generated new TSK 0
I20260808 15:45:24.820195 7076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:45:24.896087 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:25.240141 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:25.240423 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:25.240629 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:25.248497 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:25.248624 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:25.248805 7088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:25.258716 7088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:25.259977 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:25.261286 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:25.275220 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:25.275252 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:25.275827 7088 server_base.cc:1034] running on GCE node
W20260808 15:45:25.275894 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:25.276625 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:25.277603 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:25.278843 7088 hybrid_clock.cc:648] HybridClock initialized: now 1786203925278773 us; error 60 us; skew 500 ppm
I20260808 15:45:25.279193 7088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:25 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:25.282541 7088 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:25.283917 7088 webserver.cc:460] Webserver started at http://127.23.130.1:42265/ using document root <none> and password file <none>
I20260808 15:45:25.284564 7088 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:25.284665 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:25.284907 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:25.287814 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "6d021ce4625a49b0b83ac724b3924072"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "d7057108be65a35b256c4c273ff7775a"
server_key_iv: "71313d4bfc8f044eaf054acd0632635b"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.288549 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "6d021ce4625a49b0b83ac724b3924072"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "d7057108be65a35b256c4c273ff7775a"
server_key_iv: "71313d4bfc8f044eaf054acd0632635b"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.293859 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:45:25.297425 7105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.298871 7088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:45:25.299052 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "6d021ce4625a49b0b83ac724b3924072"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "d7057108be65a35b256c4c273ff7775a"
server_key_iv: "71313d4bfc8f044eaf054acd0632635b"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.299252 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:25.312976 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:25.316851 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:25.317145 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:25.318156 7088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:25.319775 7088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:25.319849 7088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.319931 7088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:25.319981 7088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.353989 7088 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:45071
I20260808 15:45:25.354043 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:45071 every 8 connection(s)
I20260808 15:45:25.355309 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:25.363139 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7088
I20260808 15:45:25.363430 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:45:25.364135 24072 external_mini_cluster.cc:1428] Setting key fd2f5b22944f89710f46660d15dd5d70
Aug 08 15:45:25 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:25.377002 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:25.357908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:39787 (local address 127.23.130.62:41991)
0808 15:45:25.358462 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:25.358469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:25.359317 (+ 848us) server_negotiation.cc:407] Connection header received
0808 15:45:25.361145 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:25.361151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:25.361246 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:25.361370 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:25.363354 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.364216 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.366214 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.366680 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.366782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:25.371177 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:25.371182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:25.371185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:25.371217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:25.373760 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:25.374527 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:25.374533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:25.374539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:25.374666 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:25.375111 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:25.375116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:25.375119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:25.375391 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:25.375561 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:25.376404 (+ 843us) server_negotiation.cc:299] Negotiation successful
0808 15:45:25.376628 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":271,"threads_started":1}
I20260808 15:45:25.378021 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:25.358352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:39787)
0808 15:45:25.359149 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:25.359189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:25.360868 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:25.361651 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:25.361660 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:25.362225 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:25.363101 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:25.363132 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.364386 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.364391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:25.366047 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:25.366060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.366911 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.366920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:25.367273 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:25.368091 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:25.368126 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:25.370961 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:25.373906 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:25.373914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:25.373930 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:25.374356 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:25.374780 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:25.374785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:25.374789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:25.374987 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:25.375532 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:25.375541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:25.375989 (+ 448us) client_negotiation.cc:769] Sending connection context
0808 15:45:25.376273 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 15:45:25.376814 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":278,"threads_started":1}
I20260808 15:45:25.379766 7228 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:25.380098 7228 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:25.380844 7228 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:25.383263 7025 ts_manager.cc:194] Registered new tserver with Master: 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071)
I20260808 15:45:25.385468 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:39787
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:45:25.429674 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:25.770637 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:25.771023 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:25.771235 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:25.778688 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:25.778817 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:25.779012 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:25.788559 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:25.789844 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:25.791230 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:25.805248 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:25.805282 7243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:25.805202 7245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:25.806123 7236 server_base.cc:1034] running on GCE node
I20260808 15:45:25.806877 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:25.807840 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:25.809049 7236 hybrid_clock.cc:648] HybridClock initialized: now 1786203925809000 us; error 50 us; skew 500 ppm
I20260808 15:45:25.809396 7236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:25 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:25.812590 7236 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:25.814056 7236 webserver.cc:460] Webserver started at http://127.23.130.2:46791/ using document root <none> and password file <none>
I20260808 15:45:25.814785 7236 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:25.814929 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:25.815236 7236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:25.818128 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "b6dcf77b19a245b5b778c89824945586"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "f8dca9e09a41f10daee8f2c621961a2b"
server_key_iv: "3f9e96ccabf98cea0ba5ba45f622b0e1"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.818984 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "b6dcf77b19a245b5b778c89824945586"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "f8dca9e09a41f10daee8f2c621961a2b"
server_key_iv: "3f9e96ccabf98cea0ba5ba45f622b0e1"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.824091 7236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:45:25.827528 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.828943 7236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:45:25.829118 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "b6dcf77b19a245b5b778c89824945586"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "f8dca9e09a41f10daee8f2c621961a2b"
server_key_iv: "3f9e96ccabf98cea0ba5ba45f622b0e1"
server_key_version: "encryptionkey@0"
I20260808 15:45:25.829350 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:25.843454 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:25.847847 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:25.848142 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:25.849047 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:25.850369 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:25.850508 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.850675 7236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:25.850730 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:25.886214 7236 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:33853
I20260808 15:45:25.886377 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:33853 every 8 connection(s)
I20260808 15:45:25.887614 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:25.896762 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7236
I20260808 15:45:25.896978 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:45:25.897573 24072 external_mini_cluster.cc:1428] Setting key d2f683cab06bdb2784c2d8ec0bbc3001
Aug 08 15:45:25 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:25.909793 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:25.890232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:53461 (local address 127.23.130.62:41991)
0808 15:45:25.890734 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:25.890739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:25.891691 (+ 952us) server_negotiation.cc:407] Connection header received
0808 15:45:25.893564 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:25.893570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:25.893663 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:25.893783 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:25.895385 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.896097 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.898505 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.899027 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.899125 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:25.902865 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:25.902874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:25.902881 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:25.902925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:25.906539 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:25.907270 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:25.907278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:25.907284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:25.907449 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:25.907862 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:25.907869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:25.907875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:25.908260 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:25.908425 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:25.909058 (+ 633us) server_negotiation.cc:299] Negotiation successful
0808 15:45:25.909319 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
I20260808 15:45:25.910480 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:25.890751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:53461)
0808 15:45:25.891529 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:25.891582 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:25.893291 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:25.894028 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:25.894036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:25.894623 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:25.895195 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:25.895212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.896266 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.896270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:25.898277 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:25.898296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:25.899260 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:25.899268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:25.899610 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:25.900367 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:25.900390 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:25.902640 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:25.906695 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:25.906701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:25.906714 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:25.907133 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:25.907557 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:25.907561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:25.907563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:25.907738 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:25.908380 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:25.908385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:25.908693 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 15:45:25.908930 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 15:45:25.909319 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260808 15:45:25.912139 7375 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:25.912482 7375 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:25.913355 7375 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:25.915063 7025 ts_manager.cc:194] Registered new tserver with Master: b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:25.916412 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:53461
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:45:25.962795 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:26.314720 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:26.315002 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:26.315202 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:26.322606 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:26.322710 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:26.322937 7381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:26.332681 7381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:26.334205 7381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:26.335474 7381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:26.348402 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:26.348461 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:26.348647 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:26.349256 7381 server_base.cc:1034] running on GCE node
I20260808 15:45:26.349843 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:26.350893 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:26.352114 7381 hybrid_clock.cc:648] HybridClock initialized: now 1786203926352072 us; error 57 us; skew 500 ppm
I20260808 15:45:26.352422 7381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:26 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203926, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:26.355715 7381 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:26.357250 7381 webserver.cc:460] Webserver started at http://127.23.130.3:45529/ using document root <none> and password file <none>
I20260808 15:45:26.357925 7381 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:26.358050 7381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:26.358352 7381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:26.361217 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "90daf8b5d9bf4d5e83145da04bf1871a"
format_stamp: "Formatted at 2026-08-08 15:45:26 on dist-test-slave-gp6n"
server_key: "aab5abebadb5df42b8801cb4b0fd17da"
server_key_iv: "2ed07d15b9fde091bc7259870ad2393a"
server_key_version: "encryptionkey@0"
I20260808 15:45:26.362038 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "90daf8b5d9bf4d5e83145da04bf1871a"
format_stamp: "Formatted at 2026-08-08 15:45:26 on dist-test-slave-gp6n"
server_key: "aab5abebadb5df42b8801cb4b0fd17da"
server_key_iv: "2ed07d15b9fde091bc7259870ad2393a"
server_key_version: "encryptionkey@0"
I20260808 15:45:26.367149 7381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:45:26.370649 7398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:26.371917 7381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:45:26.372109 7381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "90daf8b5d9bf4d5e83145da04bf1871a"
format_stamp: "Formatted at 2026-08-08 15:45:26 on dist-test-slave-gp6n"
server_key: "aab5abebadb5df42b8801cb4b0fd17da"
server_key_iv: "2ed07d15b9fde091bc7259870ad2393a"
server_key_version: "encryptionkey@0"
I20260808 15:45:26.372318 7381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:26.385962 7381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:26.388521 7228 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
I20260808 15:45:26.390698 7381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:26.391013 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:26.392048 7381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:26.393477 7381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:26.393553 7381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:26.393699 7381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:26.393769 7381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:26.427721 7381 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:32991
I20260808 15:45:26.427803 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:32991 every 8 connection(s)
I20260808 15:45:26.429039 7381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:26.430133 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7381
I20260808 15:45:26.430374 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:45:26.431114 24072 external_mini_cluster.cc:1428] Setting key 809f81c1879ff56892aa369e9ad73df0
Aug 08 15:45:26 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203926, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:26.451592 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.432343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:41449 (local address 127.23.130.62:41991)
0808 15:45:26.432964 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.432973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.434079 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 15:45:26.436353 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.436357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.436431 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.436522 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:26.438037 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.439046 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.440771 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.441177 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.441276 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.445143 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:26.445148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:26.445151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:26.445184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:26.448694 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:26.449452 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:26.449456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:26.449458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:26.449577 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:26.450031 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:26.450034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:26.450036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:26.450294 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:26.450468 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.451162 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.451324 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":269,"threads_started":1}
I20260808 15:45:26.452600 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.432950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:41449)
0808 15:45:26.433888 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.433944 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.436095 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.436705 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.436714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.437254 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:26.437851 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.437871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.439216 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.439221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.440619 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.440633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.441419 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.441425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.441631 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.442455 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:26.442492 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:26.444934 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:26.448861 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:26.448867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:26.448879 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:26.449317 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:26.449705 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:26.449709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:26.449711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:26.449928 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:26.450475 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:26.450481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:26.450829 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.451075 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.451485 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":358,"threads_started":1}
I20260808 15:45:26.454144 7522 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:26.454573 7522 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:26.455423 7522 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:26.456997 7025 ts_manager.cc:194] Registered new tserver with Master: 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3:32991)
I20260808 15:45:26.457840 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:41449
I20260808 15:45:26.460604 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:45:26.478963 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.465687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.23.130.62:41991)
0808 15:45:26.465948 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.465955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.466283 (+ 328us) server_negotiation.cc:407] Connection header received
0808 15:45:26.466492 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.466496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.466561 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.466644 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:26.468171 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.469064 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.470391 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.470850 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.470944 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.473511 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:26.473516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:26.473519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:26.473551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:26.476387 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:26.477151 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:26.477155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:26.477157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:26.477275 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:26.477653 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:26.477657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:26.477658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:26.477974 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:26.478148 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.478549 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.478711 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:45:26.483114 7025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:45:26.499701 7025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:45:26.545241 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.517736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52296)
0808 15:45:26.530642 (+ 12906us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.530661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.530797 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.531898 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.531909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.531945 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.532492 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.532510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.537342 (+ 4832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.537347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.540303 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.540324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.544018 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.544026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.544292 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.544355 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.544507 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.544747 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12753,"thread_start_us":13437,"threads_started":1}
I20260808 15:45:26.547043 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.518541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.23.130.1:45071)
0808 15:45:26.527052 (+ 8511us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.527065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.530951 (+ 3886us) server_negotiation.cc:407] Connection header received
0808 15:45:26.531083 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.531094 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.531374 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.531578 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.533211 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.535320 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.540560 (+ 5240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.543381 (+ 2821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.543546 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.543858 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.546519 (+ 2661us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.546722 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8266,"thread_start_us":8152,"threads_started":1}
I20260808 15:45:26.548444 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.515173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:32991 (local address 127.0.0.1:36166)
0808 15:45:26.529035 (+ 13862us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.529087 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.529306 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.536611 (+ 7305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.536618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.536650 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.537098 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.537114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.540053 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.540063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.543962 (+ 3899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.543987 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.547475 (+ 3488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.547485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.547768 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.547810 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.547915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.548075 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13535,"thread_start_us":14341,"threads_started":1}
I20260808 15:45:26.549345 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.515501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.23.130.3:32991)
0808 15:45:26.535675 (+ 20174us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.535687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.535719 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:45:26.535818 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.535827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.536087 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.536277 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.537341 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.539798 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.544240 (+ 4442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.546710 (+ 2470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.546891 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.547232 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.548710 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.548899 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19934,"thread_start_us":17316,"threads_started":1}
I20260808 15:45:26.552066 7156 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 (DEFAULT_TABLE table=test-table [id=25b630532d6e4f509f4ce54f94a9f1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:26.553709 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab9dd48d9f4f8fa7271c995a2b8510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:26.553746 7450 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 (DEFAULT_TABLE table=test-table [id=25b630532d6e4f509f4ce54f94a9f1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:26.554653 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.536043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33853 (local address 127.0.0.1:49280)
0808 15:45:26.545863 (+ 9820us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.545886 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.546009 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.547033 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.547041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.547069 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.547948 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.547966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.549996 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.550003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.551853 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.551869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.553921 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.553928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.554096 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.554141 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.554229 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.554373 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9643,"thread_start_us":11567,"threads_started":1}
I20260808 15:45:26.555378 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.532022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.23.130.2:33853)
0808 15:45:26.538824 (+ 6802us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.538833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.546152 (+ 7319us) server_negotiation.cc:407] Connection header received
0808 15:45:26.546281 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.546291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.546568 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.546764 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.548151 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.549789 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.552046 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.553493 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.553628 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.553821 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.554866 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.555038 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6637,"thread_start_us":6513,"threads_started":1}
I20260808 15:45:26.555481 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab9dd48d9f4f8fa7271c995a2b8510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:26.558930 7302 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 (DEFAULT_TABLE table=test-table [id=25b630532d6e4f509f4ce54f94a9f1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:26.560016 7302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab9dd48d9f4f8fa7271c995a2b8510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:26.581027 7541 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Bootstrap starting.
I20260808 15:45:26.582297 7542 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Bootstrap starting.
I20260808 15:45:26.584559 7543 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Bootstrap starting.
I20260808 15:45:26.586932 7541 tablet_bootstrap.cc:654] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:26.588150 7543 tablet_bootstrap.cc:654] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:26.588399 7541 log.cc:824] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:26.589563 7543 log.cc:824] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:26.591735 7542 tablet_bootstrap.cc:654] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:26.593395 7542 log.cc:824] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:26.598347 7541 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: No bootstrap required, opened a new log
I20260808 15:45:26.598834 7541 ts_tablet_manager.cc:1397] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260808 15:45:26.607051 7542 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: No bootstrap required, opened a new log
I20260808 15:45:26.607529 7542 ts_tablet_manager.cc:1397] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.000s
I20260808 15:45:26.609030 7543 tablet_bootstrap.cc:492] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: No bootstrap required, opened a new log
I20260808 15:45:26.609387 7543 ts_tablet_manager.cc:1397] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.018s
I20260808 15:45:26.612390 7541 raft_consensus.cc:348] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.612896 7541 raft_consensus.cc:374] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:26.613034 7541 raft_consensus.cc:729] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dcf77b19a245b5b778c89824945586, State: Initialized, Role: FOLLOWER
I20260808 15:45:26.613798 7541 consensus_queue.cc:260] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.621733 7543 raft_consensus.cc:348] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.622380 7543 raft_consensus.cc:374] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:26.622572 7543 raft_consensus.cc:729] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d021ce4625a49b0b83ac724b3924072, State: Initialized, Role: FOLLOWER
I20260808 15:45:26.623382 7542 raft_consensus.cc:348] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.623502 7543 consensus_queue.cc:260] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.623982 7542 raft_consensus.cc:374] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:26.624167 7542 raft_consensus.cc:729] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90daf8b5d9bf4d5e83145da04bf1871a, State: Initialized, Role: FOLLOWER
I20260808 15:45:26.624868 7375 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
I20260808 15:45:26.625021 7542 consensus_queue.cc:260] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.625375 7541 ts_tablet_manager.cc:1428] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Time spent starting tablet: real 0.026s user 0.017s sys 0.006s
I20260808 15:45:26.630530 7543 ts_tablet_manager.cc:1428] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Time spent starting tablet: real 0.021s user 0.015s sys 0.003s
I20260808 15:45:26.633121 7542 ts_tablet_manager.cc:1428] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Time spent starting tablet: real 0.025s user 0.020s sys 0.004s
I20260808 15:45:26.633582 7522 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
W20260808 15:45:26.639613 7376 tablet.cc:2369] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:45:26.681039 7523 tablet.cc:2369] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:45:26.857853 7229 tablet.cc:2369] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:26.885823 7549 raft_consensus.cc:482] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:45:26.886199 7549 raft_consensus.cc:504] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.887758 7547 raft_consensus.cc:482] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:45:26.888063 7547 raft_consensus.cc:504] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.888340 7549 leader_election.cc:288] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071), b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:26.890167 7547 leader_election.cc:288] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3:32991), 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071)
I20260808 15:45:26.898136 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.888676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.23.130.3:50279)
0808 15:45:26.888907 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.888930 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.889075 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.889500 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.889505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.889532 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.889761 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.889773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.892142 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.892148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.894542 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.894564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.897076 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.897085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.897359 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.897414 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.897522 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.897720 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:45:26.899026 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.888961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:50279 (local address 127.23.130.1:45071)
0808 15:45:26.889181 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.889187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.889208 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:45:26.889304 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.889310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.889384 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.889476 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.889953 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.891136 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.894807 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.896626 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.896826 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.896939 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.898585 (+ 1646us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.898747 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:45:26.910245 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.889644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:60309 (local address 127.23.130.2:33853)
0808 15:45:26.889854 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.889860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.890083 (+ 223us) server_negotiation.cc:407] Connection header received
0808 15:45:26.890353 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.890358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.890453 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.890530 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.891323 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.892528 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.904152 (+ 11624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.906027 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.906209 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.906315 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.909760 (+ 3445us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.909914 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:45:26.910063 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d021ce4625a49b0b83ac724b3924072" is_pre_election: true
I20260808 15:45:26.910676 7178 raft_consensus.cc:2454] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 0.
I20260808 15:45:26.911093 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.889073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33853 (local address 127.23.130.3:60309)
0808 15:45:26.889987 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.890004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.890179 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.890714 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.890719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.890743 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.891132 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.891144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.898736 (+ 7592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.898741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.901205 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.901227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.906619 (+ 5392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.906628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.909568 (+ 2940us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.909608 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.910527 (+ 919us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.910692 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":2643,"thread_start_us":797,"threads_started":1}
I20260808 15:45:26.911839 7401 leader_election.cc:302] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d021ce4625a49b0b83ac724b3924072, 90daf8b5d9bf4d5e83145da04bf1871a; no voters:
I20260808 15:45:26.911975 7325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6dcf77b19a245b5b778c89824945586" is_pre_election: true
I20260808 15:45:26.912393 7549 raft_consensus.cc:2781] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:45:26.912555 7549 raft_consensus.cc:482] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:45:26.912513 7325 raft_consensus.cc:2454] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 0.
I20260808 15:45:26.912664 7549 raft_consensus.cc:3037] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:26.913890 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.890670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:36147 (local address 127.23.130.3:32991)
0808 15:45:26.890832 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.890838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.906186 (+ 15348us) server_negotiation.cc:407] Connection header received
0808 15:45:26.906650 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.906656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.906728 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.906784 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.907698 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.908741 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.911333 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.912626 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.912776 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.912846 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.913499 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.913636 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":47}
I20260808 15:45:26.915035 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.890331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:32991 (local address 127.23.130.2:36147)
0808 15:45:26.906077 (+ 15746us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.906103 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.906486 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.907010 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.907015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.907039 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.907520 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.907531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.908927 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.908931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.911130 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.911147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.913080 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.913086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.913297 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.913335 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.914111 (+ 776us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.914249 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15616,"thread_start_us":15873,"threads_started":1}
I20260808 15:45:26.916054 7549 raft_consensus.cc:504] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.916325 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.891601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.23.130.2:39947)
0808 15:45:26.908893 (+ 17292us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:26.908912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:26.909353 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:26.909866 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:26.909870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:26.909898 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:26.910312 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.910323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.911865 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.911871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.913869 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:26.913881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.915450 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.915456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:26.915706 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.915743 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:45:26.915850 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 15:45:26.915978 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17144,"thread_start_us":17413,"threads_started":1}
I20260808 15:45:26.916023 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "b6dcf77b19a245b5b778c89824945586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" is_pre_election: true
I20260808 15:45:26.916384 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.892740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:39947 (local address 127.23.130.1:45071)
0808 15:45:26.899248 (+ 6508us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.899254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.909006 (+ 9752us) server_negotiation.cc:407] Connection header received
0808 15:45:26.909531 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.909536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.909607 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.909679 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:26.910549 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.911686 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.914106 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.915225 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.915431 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.915497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.915912 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.916064 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6407,"thread_start_us":16228,"threads_started":1}
I20260808 15:45:26.916671 7472 raft_consensus.cc:2379] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6dcf77b19a245b5b778c89824945586 in current term 1: Already voted for candidate 90daf8b5d9bf4d5e83145da04bf1871a in this term.
I20260808 15:45:26.917204 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "b6dcf77b19a245b5b778c89824945586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d021ce4625a49b0b83ac724b3924072" is_pre_election: true
I20260808 15:45:26.917323 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d021ce4625a49b0b83ac724b3924072"
I20260808 15:45:26.917486 7178 raft_consensus.cc:2454] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6dcf77b19a245b5b778c89824945586 in term 0.
I20260808 15:45:26.917518 7325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab9dd48d9f4f8fa7271c995a2b8510" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6dcf77b19a245b5b778c89824945586"
I20260808 15:45:26.917721 7325 raft_consensus.cc:3037] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:26.918226 7549 leader_election.cc:288] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 election: Requested vote from peers 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071), b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:26.918159 7255 leader_election.cc:302] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d021ce4625a49b0b83ac724b3924072, b6dcf77b19a245b5b778c89824945586; no voters: 90daf8b5d9bf4d5e83145da04bf1871a
I20260808 15:45:26.920568 7325 raft_consensus.cc:2454] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 1.
I20260808 15:45:26.920807 7547 raft_consensus.cc:2741] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 15:45:26.921063 7401 leader_election.cc:302] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90daf8b5d9bf4d5e83145da04bf1871a, b6dcf77b19a245b5b778c89824945586; no voters: 6d021ce4625a49b0b83ac724b3924072
I20260808 15:45:26.921386 7549 raft_consensus.cc:2781] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:26.921900 7549 raft_consensus.cc:686] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 LEADER]: Becoming Leader. State: Replica: 90daf8b5d9bf4d5e83145da04bf1871a, State: Running, Role: LEADER
I20260808 15:45:26.922391 7549 consensus_queue.cc:237] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:26.926894 7024 catalog_manager.cc:5495] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3). New cstate: current_term: 1 leader_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } health_report { overall_health: UNKNOWN } } }
I20260808 15:45:26.986194 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:26.980285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.23.130.3:32991)
0808 15:45:26.980542 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:26.980549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:26.980816 (+ 267us) server_negotiation.cc:407] Connection header received
0808 15:45:26.981073 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:26.981085 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:26.981176 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:26.981332 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:45:26.981752 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.982602 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.984024 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:26.984437 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:26.984580 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:26.985073 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:45:26.985595 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:45:26.985759 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:26.985816 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 15:45:26.985896 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:45:26.994452 7177 raft_consensus.cc:3037] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:26.994688 7325 raft_consensus.cc:1261] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Refusing update from remote peer 90daf8b5d9bf4d5e83145da04bf1871a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:26.995611 7555 consensus_queue.cc:1035] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:26.997924 7177 raft_consensus.cc:1261] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Refusing update from remote peer 90daf8b5d9bf4d5e83145da04bf1871a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:26.998761 7555 consensus_queue.cc:1035] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:27.011055 7561 mvcc.cc:205] Tried to move back new op lower bound from 7316291284959305728 to 7316291284677967872. Current Snapshot: MvccSnapshot[applied={T|T < 7316291284959305728}]
I20260808 15:45:27.014263 7564 mvcc.cc:205] Tried to move back new op lower bound from 7316291284959305728 to 7316291284677967872. Current Snapshot: MvccSnapshot[applied={T|T < 7316291284959305728}]
I20260808 15:45:27.023010 7025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:27.023324 7025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:27.026942 7025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Sending DeleteTablet for 3 replicas of tablet 37ab9dd48d9f4f8fa7271c995a2b8510
I20260808 15:45:27.028033 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:27 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:36166
I20260808 15:45:27.028097 7302 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:27 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:49280
I20260808 15:45:27.028277 7156 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab9dd48d9f4f8fa7271c995a2b8510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:27 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:52296
I20260808 15:45:27.029350 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7088
I20260808 15:45:27.046058 7571 tablet_replica.cc:330] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: stopping tablet replica
I20260808 15:45:27.046744 7572 tablet_replica.cc:330] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: stopping tablet replica
I20260808 15:45:27.047338 7572 raft_consensus.cc:2229] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:27.047919 7571 raft_consensus.cc:2229] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:45:27.048602 7572 raft_consensus.cc:2258] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:27.049387 7571 raft_consensus.cc:2258] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:27.050230 7570 tablet_replica.cc:330] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: stopping tablet replica
I20260808 15:45:27.050688 7570 raft_consensus.cc:2229] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:27.051200 7570 raft_consensus.cc:2258] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:27.053311 7571 ts_tablet_manager.cc:1905] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:27.053452 7570 ts_tablet_manager.cc:1905] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:45:27 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:45:27.054392 7572 ts_tablet_manager.cc:1905] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:27.060810 7570 ts_tablet_manager.cc:1918] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:27.061043 7570 log.cc:1196] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/37ab9dd48d9f4f8fa7271c995a2b8510
I20260808 15:45:27.061650 7570 ts_tablet_manager.cc:1939] T 37ab9dd48d9f4f8fa7271c995a2b8510 P b6dcf77b19a245b5b778c89824945586: Deleting consensus metadata
I20260808 15:45:27.063978 7011 catalog_manager.cc:4841] TS b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853): tablet 37ab9dd48d9f4f8fa7271c995a2b8510 (table test-table [id=25b630532d6e4f509f4ce54f94a9f1c1]) successfully deleted
I20260808 15:45:27.064831 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.030894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.23.130.1:45071)
0808 15:45:27.031124 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:27.031134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:27.038030 (+ 6896us) server_negotiation.cc:407] Connection header received
0808 15:45:27.039134 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:27.039143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:27.039281 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:27.039371 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:27.044354 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.045835 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.047978 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.048631 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.048743 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:27.055363 (+ 6620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:27.055379 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:27.055401 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:27.055465 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:27.059890 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.060911 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.060925 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.060929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.061111 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.061618 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.061623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.061627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.062167 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:27.062574 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:27.063399 (+ 825us) server_negotiation.cc:299] Negotiation successful
0808 15:45:27.064417 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:45:27.066591 7222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:27.067349 7571 ts_tablet_manager.cc:1918] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:27.066570 7572 ts_tablet_manager.cc:1918] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:27.067580 7571 log.cc:1196] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/37ab9dd48d9f4f8fa7271c995a2b8510
I20260808 15:45:27.067615 7572 log.cc:1196] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/37ab9dd48d9f4f8fa7271c995a2b8510
I20260808 15:45:27.068179 7571 ts_tablet_manager.cc:1939] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting consensus metadata
I20260808 15:45:27.070611 7010 catalog_manager.cc:4841] TS 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3:32991): tablet 37ab9dd48d9f4f8fa7271c995a2b8510 (table test-table [id=25b630532d6e4f509f4ce54f94a9f1c1]) successfully deleted
I20260808 15:45:27.176173 7572 ts_tablet_manager.cc:1939] T 37ab9dd48d9f4f8fa7271c995a2b8510 P 6d021ce4625a49b0b83ac724b3924072: Deleting consensus metadata
I20260808 15:45:27.176920 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7088
W20260808 15:45:27.197849 7011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:45071 (error 108)
I20260808 15:45:27.198293 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7236
I20260808 15:45:27.200387 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.199497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52320)
0808 15:45:27.200018 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.200154 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":260,"threads_started":1}
W20260808 15:45:27.201634 7011 catalog_manager.cc:4584] TS 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37ab9dd48d9f4f8fa7271c995a2b8510: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Aug 08 15:45:27 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:45:27.219666 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.199143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.23.130.2:33853)
0808 15:45:27.199325 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:27.199331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:27.199379 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:45:27.199609 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:27.199615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:27.199707 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:27.199825 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:27.201501 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.202365 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.203848 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.204326 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.204468 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:27.213386 (+ 8918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:27.213402 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:27.213417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:27.213456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:27.216635 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.217371 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.217381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.217384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.217526 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.217929 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.217936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.217943 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.218335 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:27.218729 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:27.219064 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 15:45:27.219313 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:45:27.220347 7369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:27.255410 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.254872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52326)
0808 15:45:27.255117 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.255229 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:45:27.256654 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.256214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52332)
0808 15:45:27.256380 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.256489 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 15:45:27.315111 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7236
I20260808 15:45:27.317514 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.317008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52348)
0808 15:45:27.317298 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.317385 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260808 15:45:27.318578 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.318165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52350)
0808 15:45:27.318301 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.318379 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 15:45:27.341594 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7381
Aug 08 15:45:27 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:45:27.360122 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.342490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.23.130.3:32991)
0808 15:45:27.342695 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:27.342699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:27.342772 (+ 73us) server_negotiation.cc:407] Connection header received
0808 15:45:27.342975 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:27.342979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:27.343039 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:27.343132 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:27.344893 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.345671 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.347785 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:27.348249 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:27.348345 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:27.353835 (+ 5490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:27.353849 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:27.353865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:27.353901 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:27.357216 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.357933 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.357940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.357943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.358071 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:27.358502 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:27.358506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:27.358508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:27.358898 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:27.359279 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:27.359535 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 15:45:27.359777 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:45:27.360801 7515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:27.399883 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.399365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52356)
0808 15:45:27.399633 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.399727 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 15:45:27.401095 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.400610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52366)
0808 15:45:27.400845 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.400946 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 15:45:27.486901 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7381
I20260808 15:45:27.506347 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 6993
I20260808 15:45:27.507604 7057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:27.618261 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:27.617460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52382)
0808 15:45:27.617878 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:27.618009 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 15:45:27.618552 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 6993
I20260808 15:45:27.640851 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--webserver_interface=127.23.130.62
--webserver_port=40319
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:41991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:27.982532 7580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:27.982817 7580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:27.982909 7580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:27.990020 7580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:27.990120 7580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:27.990173 7580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:27.990231 7580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:27.990275 7580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:28.000120 7580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:41991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=40319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:28.001332 7580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:28.002640 7580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:28.028676 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:28.028647 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:28.028704 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:28.029692 7580 server_base.cc:1034] running on GCE node
I20260808 15:45:28.030642 7580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:28.032286 7580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:28.033506 7580 hybrid_clock.cc:648] HybridClock initialized: now 1786203928033458 us; error 30 us; skew 500 ppm
I20260808 15:45:28.033890 7580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:28 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:28.042856 7580 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:28.044611 7580 webserver.cc:460] Webserver started at http://127.23.130.62:40319/ using document root <none> and password file <none>
I20260808 15:45:28.045403 7580 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:28.045545 7580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:28.051923 7580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 15:45:28.055275 7599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:28.056808 7580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:45:28.056969 7580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "b8c5a7613d44459198cd4935c67984a4"
format_stamp: "Formatted at 2026-08-08 15:45:24 on dist-test-slave-gp6n"
server_key: "16c453475c8e9e499ba96c013c15cd59"
server_key_iv: "c1d6b95fd59858a9f4dd47c6a668cffb"
server_key_version: "encryptionkey@0"
I20260808 15:45:28.058039 7580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:28.094878 7580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:28.099331 7580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:28.099581 7580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:28.119618 7580 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:41991
I20260808 15:45:28.119637 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:41991 every 8 connection(s)
I20260808 15:45:28.121188 7580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:28.128489 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7580
I20260808 15:45:28.128607 7656 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:45:28.129580 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:45071
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=42265
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:28.133890 7656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap starting.
I20260808 15:45:28.156919 7656 log.cc:824] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:28.170377 7656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:45:28.171078 7656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap complete.
I20260808 15:45:28.180892 7656 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:28.181738 7656 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Initialized, Role: FOLLOWER
I20260808 15:45:28.182390 7656 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:28.182716 7656 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:28.182889 7656 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:28.183096 7656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:45:28.187019 7656 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:28.187577 7656 leader_election.cc:302] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c5a7613d44459198cd4935c67984a4; no voters:
I20260808 15:45:28.188280 7656 leader_election.cc:288] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:45:28.188431 7661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:45:28.189766 7661 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 LEADER]: Becoming Leader. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Running, Role: LEADER
I20260808 15:45:28.190439 7661 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:28.190809 7656 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:28.193226 7663 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c5a7613d44459198cd4935c67984a4. Latest consensus state: current_term: 2 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:28.193626 7663 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:28.193446 7662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:28.193864 7662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:28.198348 7668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:28.208092 7668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:28.209991 7668 catalog_manager.cc:1259] Loaded cluster ID: 93d36f4984694822b871be7dd019848a
I20260808 15:45:28.210182 7668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:28.214533 7668 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 15:45:28.232542 7682 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:45:28.237435 7668 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 15:45:28.244937 7668 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:28.246294 7668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Loaded TSK: 0
I20260808 15:45:28.247475 7668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:45:28.516712 7658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:28.517047 7658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:28.517223 7658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:28.524411 7658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:28.524561 7658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:28.524755 7658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:28.534381 7658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:45071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=42265
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:28.535672 7658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:28.536988 7658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:28.551623 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:28.552232 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:28.552428 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:28.552778 7658 server_base.cc:1034] running on GCE node
I20260808 15:45:28.553588 7658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:28.554925 7658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:28.556195 7658 hybrid_clock.cc:648] HybridClock initialized: now 1786203928556121 us; error 53 us; skew 500 ppm
I20260808 15:45:28.556587 7658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:28 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:28.560811 7658 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:28.562790 7658 webserver.cc:460] Webserver started at http://127.23.130.1:42265/ using document root <none> and password file <none>
I20260808 15:45:28.563767 7658 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:28.563946 7658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:28.571164 7658 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:45:28.574620 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:28.576198 7658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:45:28.576360 7658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "6d021ce4625a49b0b83ac724b3924072"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "d7057108be65a35b256c4c273ff7775a"
server_key_iv: "71313d4bfc8f044eaf054acd0632635b"
server_key_version: "encryptionkey@0"
I20260808 15:45:28.577648 7658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:28.602888 7658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:28.608249 7658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:28.608539 7658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:28.609736 7658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:28.611388 7658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:28.611498 7658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:28.611610 7658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:28.611716 7658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:28.649808 7658 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:45071
I20260808 15:45:28.649871 7820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:45071 every 8 connection(s)
I20260808 15:45:28.651443 7658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:28.662025 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7658
I20260808 15:45:28.662827 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:33853
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=46791
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:28 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:28.678640 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:28.654674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:51071 (local address 127.23.130.62:41991)
0808 15:45:28.655318 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:28.655330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:28.655858 (+ 528us) server_negotiation.cc:407] Connection header received
0808 15:45:28.657972 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:28.657986 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:28.658308 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:28.658686 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:28.660223 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:28.661707 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:28.663740 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:28.664896 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:28.665115 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:28.669019 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:28.669034 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:28.669058 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:28.669119 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:28.673233 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:28.674168 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:28.674176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:28.674182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:28.674390 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:28.674943 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:28.674949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:28.674960 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:28.675484 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:28.675704 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:28.676619 (+ 915us) server_negotiation.cc:299] Negotiation successful
0808 15:45:28.677075 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
I20260808 15:45:28.678640 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:28.654795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:51071)
0808 15:45:28.655676 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:28.655728 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:28.657474 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:28.658866 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:28.658876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:28.659475 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:28.660014 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:28.660031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:28.661969 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:28.661973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:28.663591 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:28.663603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:28.665138 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:28.665143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:28.665342 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:28.666149 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:28.666173 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:28.668720 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:28.673420 (+ 4700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:28.673427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:28.673443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:28.673961 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:28.674596 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:28.674601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:28.674603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:28.674805 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:28.675651 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:28.675660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:28.676047 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 15:45:28.676361 (+ 314us) client_negotiation.cc:240] Negotiation successful
0808 15:45:28.677021 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":369,"threads_started":1}
I20260808 15:45:28.681804 7821 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:28.682192 7821 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:28.683197 7821 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:28.685609 7616 ts_manager.cc:194] Registered new tserver with Master: 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071)
I20260808 15:45:28.687317 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:51071
W20260808 15:45:29.010967 7826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:29.011260 7826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:29.011484 7826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:29.018916 7826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:29.019026 7826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:29.019209 7826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:29.029322 7826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=46791
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:29.030699 7826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:29.032059 7826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:29.045746 7832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:29.045814 7831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:29.045849 7834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:29.046427 7826 server_base.cc:1034] running on GCE node
I20260808 15:45:29.047236 7826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:29.048249 7826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:29.049479 7826 hybrid_clock.cc:648] HybridClock initialized: now 1786203929049430 us; error 45 us; skew 500 ppm
I20260808 15:45:29.049809 7826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:29 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:29.053138 7826 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:29.054961 7826 webserver.cc:460] Webserver started at http://127.23.130.2:46791/ using document root <none> and password file <none>
I20260808 15:45:29.055658 7826 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:29.055826 7826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:29.061270 7826 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:45:29.064231 7842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.065668 7826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:45:29.065866 7826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "b6dcf77b19a245b5b778c89824945586"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "f8dca9e09a41f10daee8f2c621961a2b"
server_key_iv: "3f9e96ccabf98cea0ba5ba45f622b0e1"
server_key_version: "encryptionkey@0"
I20260808 15:45:29.066972 7826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:29.101172 7826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:29.105818 7826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:29.106103 7826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:29.107244 7826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:29.108597 7826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:29.108724 7826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.108866 7826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:29.108958 7826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.144456 7826 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:33853
I20260808 15:45:29.144543 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:33853 every 8 connection(s)
I20260808 15:45:29.146059 7826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:29.150949 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7826
I20260808 15:45:29.151757 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:32991
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=45529
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:45:29 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:29.170440 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.148822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:53177 (local address 127.23.130.62:41991)
0808 15:45:29.149010 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.149015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.150193 (+ 1178us) server_negotiation.cc:407] Connection header received
0808 15:45:29.153028 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.153033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.153174 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.153322 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:29.155457 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.156261 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.158913 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.159429 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.159546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.164310 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:29.164315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:29.164318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:29.164351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:29.167192 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.167957 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.167971 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.167974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.168101 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.168568 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.168572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.168574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.168859 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:29.169040 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.169771 (+ 731us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.170046 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:45:29.171511 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.149343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:53177)
0808 15:45:29.150037 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.150081 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.152697 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.153608 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.153624 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.154439 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:29.155253 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.155280 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.156448 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.156456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.158725 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.158743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.159672 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.159681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.160008 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.160807 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:29.160843 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:29.164033 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:29.167352 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:29.167361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:29.167379 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:29.167818 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:29.168218 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:29.168223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:29.168227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:29.168427 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:29.168993 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:29.169003 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:29.169383 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.169665 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.170214 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":270,"threads_started":1}
I20260808 15:45:29.173171 7967 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:29.173487 7967 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:29.174244 7967 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:29.175863 7616 ts_manager.cc:194] Registered new tserver with Master: b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:29.177192 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:53177
W20260808 15:45:29.512156 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:29.512440 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:29.512537 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:29.519680 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:29.519784 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:29.519896 7970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:29.529446 7970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:32991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=45529
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:29.530723 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:29.532032 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:29.545934 7977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:29.545995 7978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:29.546162 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:29.546970 7970 server_base.cc:1034] running on GCE node
I20260808 15:45:29.547639 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:29.548633 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:29.549878 7970 hybrid_clock.cc:648] HybridClock initialized: now 1786203929549836 us; error 70 us; skew 500 ppm
I20260808 15:45:29.550210 7970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:29 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:29.553687 7970 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:29.555236 7970 webserver.cc:460] Webserver started at http://127.23.130.3:45529/ using document root <none> and password file <none>
I20260808 15:45:29.555930 7970 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:29.556092 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:29.561420 7970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:45:29.564425 7987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.565728 7970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:45:29.565900 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "90daf8b5d9bf4d5e83145da04bf1871a"
format_stamp: "Formatted at 2026-08-08 15:45:26 on dist-test-slave-gp6n"
server_key: "aab5abebadb5df42b8801cb4b0fd17da"
server_key_iv: "2ed07d15b9fde091bc7259870ad2393a"
server_key_version: "encryptionkey@0"
I20260808 15:45:29.566962 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:29.587450 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:29.591928 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:29.592199 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:29.593155 7970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:29.594599 7970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:29.594722 7970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.594870 7970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:29.594959 7970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:29.628721 7970 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:32991
I20260808 15:45:29.628777 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:32991 every 8 connection(s)
I20260808 15:45:29.630218 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:29.634232 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 7970
Aug 08 15:45:29 dist-test-slave-gp6n krb5kdc[6977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:29.651013 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.635313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.23.130.62:41991)
0808 15:45:29.635793 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.635801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.636212 (+ 411us) server_negotiation.cc:407] Connection header received
0808 15:45:29.636370 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.636376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.636457 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.636587 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:29.638273 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.639205 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.640844 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.641499 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.641589 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.644529 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:29.644533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:29.644537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:29.644566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:29.647598 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.648549 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.648553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.648555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.648681 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.649195 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.649200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.649203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.649647 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:29.649885 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.650304 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.650590 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":262,"threads_started":1}
I20260808 15:45:29.653932 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.632975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59699 (local address 127.23.130.62:41991)
0808 15:45:29.633154 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.633159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.634491 (+ 1332us) server_negotiation.cc:407] Connection header received
0808 15:45:29.636859 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.636865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.636937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.637044 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:29.638673 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.639581 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.641361 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.642025 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.642169 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.645958 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:29.645963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:29.645966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:29.646013 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:29.649384 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.650302 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.650308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.650313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.650471 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:29.651012 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:29.651018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:29.651021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:29.651401 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:29.651544 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.652541 (+ 997us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.653705 (+ 1164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:45:29.654191 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.633407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:59699)
0808 15:45:29.634285 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.634347 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.636547 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.637258 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.637266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.637768 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:29.638480 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.638506 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.639756 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.639760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.641210 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.641226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.642337 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.642350 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.642642 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.643240 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:29.643266 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:29.645742 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:29.649524 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:29.649533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:29.649548 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:29.650137 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:29.650617 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:29.650623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:29.650627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:29.650868 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:29.651544 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:29.651551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:29.651969 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.652287 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.652800 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260808 15:45:29.655654 8114 heartbeater.cc:347] Connected to a master server at 127.23.130.62:41991
I20260808 15:45:29.655972 8114 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:29.656684 8114 heartbeater.cc:511] Master 127.23.130.62:41991 requested a full tablet report, sending...
I20260808 15:45:29.658016 7616 ts_manager.cc:194] Registered new tserver with Master: 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3:32991)
I20260808 15:45:29.658895 7616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:59699
I20260808 15:45:29.665793 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:45:29.672691 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.666740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.23.130.62:41991)
0808 15:45:29.666903 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.666907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.667400 (+ 493us) server_negotiation.cc:407] Connection header received
0808 15:45:29.667570 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.667573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.667703 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.667819 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:45:29.668250 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.668961 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.670459 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.670915 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.671048 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.671632 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:45:29.672120 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:45:29.672266 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.672347 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.672423 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:45:29.673722 7616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:45:29.681030 7616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:45:29.691442 7821 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
I20260808 15:45:29.707733 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.693002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:32991 (local address 127.0.0.1:36198)
0808 15:45:29.693737 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.693787 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.693977 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.699063 (+ 5086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.699070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.699106 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:29.699601 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.699615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.702095 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.702101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.704762 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.704783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.706716 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.706723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.706917 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.706974 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.707090 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.707276 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"thread_start_us":846,"threads_started":1}
I20260808 15:45:29.709075 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.693002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52388)
0808 15:45:29.698932 (+ 5930us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.698955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.699161 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.700067 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.700072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.700099 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:29.700476 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.700488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.703543 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.703553 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.705632 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.705649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.708178 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.708186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.708382 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.708418 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.708553 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.708695 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"mutex_wait_us":519,"thread_start_us":5093,"threads_started":1}
I20260808 15:45:29.709136 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.693082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.23.130.1:45071)
0808 15:45:29.694640 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.694652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.699053 (+ 4401us) server_negotiation.cc:407] Connection header received
0808 15:45:29.699361 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.699369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.699600 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.699757 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:29.700700 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.702276 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.705812 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.707331 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.707450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.707677 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.708611 (+ 934us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.708785 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1372,"thread_start_us":1208,"threads_started":1}
I20260808 15:45:29.710103 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.693627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.23.130.3:32991)
0808 15:45:29.698274 (+ 4647us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.698283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.698316 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:45:29.698463 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.698471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.698821 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.699850 (+ 1029us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:29.700119 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.701941 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.704984 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.706486 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.709404 (+ 2918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.709647 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.709720 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.709859 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4462,"thread_start_us":4314,"threads_started":1}
I20260808 15:45:29.713451 8040 tablet_service.cc:1467] Processing CreateTablet for tablet b7719c04877c478abf3cadb591e97969 (DEFAULT_TABLE table=test-table [id=aa4d794561b54b468df26cb63573386f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:29.713593 7751 tablet_service.cc:1467] Processing CreateTablet for tablet b7719c04877c478abf3cadb591e97969 (DEFAULT_TABLE table=test-table [id=aa4d794561b54b468df26cb63573386f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:29.714985 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7719c04877c478abf3cadb591e97969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:29.715364 7751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7719c04877c478abf3cadb591e97969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:29.716039 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.699027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33853 (local address 127.0.0.1:49304)
0808 15:45:29.707992 (+ 8965us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.708017 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.708118 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.709372 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.709381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.709409 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:29.709574 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.709582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.711610 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.711615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.713394 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.713409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.715299 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.715308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.715495 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.715531 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.715599 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.715739 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8858,"thread_start_us":10084,"threads_started":1}
I20260808 15:45:29.717118 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.699279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.23.130.2:33853)
0808 15:45:29.708583 (+ 9304us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.708592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.708620 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:45:29.708714 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.708721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.708953 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.709781 (+ 828us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:29.710054 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.711444 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.713593 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.715106 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.716380 (+ 1274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.716568 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.716640 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.716827 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9136,"thread_start_us":9038,"threads_started":1}
I20260808 15:45:29.720829 7895 tablet_service.cc:1467] Processing CreateTablet for tablet b7719c04877c478abf3cadb591e97969 (DEFAULT_TABLE table=test-table [id=aa4d794561b54b468df26cb63573386f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:29.722223 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7719c04877c478abf3cadb591e97969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:29.728412 8131 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Bootstrap starting.
I20260808 15:45:29.731495 8131 tablet_bootstrap.cc:654] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:29.732519 8131 log.cc:824] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:29.733969 8132 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Bootstrap starting.
I20260808 15:45:29.736105 8131 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: No bootstrap required, opened a new log
I20260808 15:45:29.736418 8131 ts_tablet_manager.cc:1397] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:45:29.737679 8132 tablet_bootstrap.cc:654] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:29.738241 8134 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Bootstrap starting.
I20260808 15:45:29.739032 8132 log.cc:824] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:29.741254 8134 tablet_bootstrap.cc:654] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:29.742594 8134 log.cc:824] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:29.744129 8132 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: No bootstrap required, opened a new log
I20260808 15:45:29.744407 8132 ts_tablet_manager.cc:1397] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:45:29.745121 8134 tablet_bootstrap.cc:492] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: No bootstrap required, opened a new log
I20260808 15:45:29.745383 8134 ts_tablet_manager.cc:1397] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 15:45:29.747421 8131 raft_consensus.cc:348] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.747859 8131 raft_consensus.cc:374] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:29.748031 8131 raft_consensus.cc:729] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90daf8b5d9bf4d5e83145da04bf1871a, State: Initialized, Role: FOLLOWER
I20260808 15:45:29.748667 8131 consensus_queue.cc:260] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.749842 8114 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
I20260808 15:45:29.750494 8131 ts_tablet_manager.cc:1428] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260808 15:45:29.753760 8132 raft_consensus.cc:348] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.754128 8132 raft_consensus.cc:374] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:29.754245 8132 raft_consensus.cc:729] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d021ce4625a49b0b83ac724b3924072, State: Initialized, Role: FOLLOWER
I20260808 15:45:29.754925 8132 consensus_queue.cc:260] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.756639 8132 ts_tablet_manager.cc:1428] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260808 15:45:29.757629 8134 raft_consensus.cc:348] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.757992 8134 raft_consensus.cc:374] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:29.758119 8134 raft_consensus.cc:729] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dcf77b19a245b5b778c89824945586, State: Initialized, Role: FOLLOWER
I20260808 15:45:29.758697 8134 consensus_queue.cc:260] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.760236 7967 heartbeater.cc:503] Master 127.23.130.62:41991 was elected leader, sending a full tablet report...
I20260808 15:45:29.760447 8134 ts_tablet_manager.cc:1428] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260808 15:45:29.881901 8115 tablet.cc:2369] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:45:29.898324 7968 tablet.cc:2369] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:29.903442 8138 raft_consensus.cc:482] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 15:45:29.903996 7822 tablet.cc:2369] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:29.903766 8138 raft_consensus.cc:504] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.905716 8138 leader_election.cc:288] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071), b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:29.911588 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.906294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59115 (local address 127.23.130.1:45071)
0808 15:45:29.906558 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.906565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.906589 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:45:29.906672 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.906680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.906783 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.906909 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:29.907449 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.908173 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.909865 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.910687 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.910866 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.910943 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.911215 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.911339 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 15:45:29.911614 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.906069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.23.130.3:59115)
0808 15:45:29.906292 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.906314 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.906497 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.907082 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.907085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.907109 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:29.907277 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.907284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.908376 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.908381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.909686 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.909698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.910841 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.910849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.911035 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.911059 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.911187 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.911320 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:45:29.912377 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.906517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33853 (local address 127.23.130.3:41881)
0808 15:45:29.907157 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:29.907170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:29.907310 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:29.907652 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:29.907656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:29.907679 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:29.907999 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.908008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.908975 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.908981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.910391 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:29.910462 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.911607 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.911614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:29.911799 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.911830 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:45:29.911969 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 15:45:29.912067 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":1,"thread_start_us":326,"threads_started":1}
I20260808 15:45:29.912413 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.906784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:41881 (local address 127.23.130.2:33853)
0808 15:45:29.906993 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.907001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.907256 (+ 255us) server_negotiation.cc:407] Connection header received
0808 15:45:29.907459 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.907465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.907533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.907631 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:29.908145 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.908826 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.910605 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.911392 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.911571 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.911643 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.912036 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.912180 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:45:29.912868 7771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7719c04877c478abf3cadb591e97969" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d021ce4625a49b0b83ac724b3924072" is_pre_election: true
I20260808 15:45:29.913472 7771 raft_consensus.cc:2454] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 0.
I20260808 15:45:29.913522 7915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7719c04877c478abf3cadb591e97969" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6dcf77b19a245b5b778c89824945586" is_pre_election: true
I20260808 15:45:29.914108 7915 raft_consensus.cc:2454] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 0.
I20260808 15:45:29.914181 7990 leader_election.cc:302] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d021ce4625a49b0b83ac724b3924072, 90daf8b5d9bf4d5e83145da04bf1871a; no voters:
I20260808 15:45:29.914629 8138 raft_consensus.cc:2781] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:45:29.914790 8138 raft_consensus.cc:482] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:45:29.914930 8138 raft_consensus.cc:3037] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:29.917140 8138 raft_consensus.cc:504] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.917701 8138 leader_election.cc:288] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 election: Requested vote from peers 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071), b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853)
I20260808 15:45:29.918246 7771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7719c04877c478abf3cadb591e97969" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d021ce4625a49b0b83ac724b3924072"
I20260808 15:45:29.918319 7915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7719c04877c478abf3cadb591e97969" candidate_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6dcf77b19a245b5b778c89824945586"
I20260808 15:45:29.918497 7771 raft_consensus.cc:3037] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:29.918632 7915 raft_consensus.cc:3037] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:29.920713 7771 raft_consensus.cc:2454] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 1.
I20260808 15:45:29.920887 7915 raft_consensus.cc:2454] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90daf8b5d9bf4d5e83145da04bf1871a in term 1.
I20260808 15:45:29.921120 7990 leader_election.cc:302] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d021ce4625a49b0b83ac724b3924072, 90daf8b5d9bf4d5e83145da04bf1871a; no voters:
I20260808 15:45:29.921444 8138 raft_consensus.cc:2781] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:29.921898 8138 raft_consensus.cc:686] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 LEADER]: Becoming Leader. State: Replica: 90daf8b5d9bf4d5e83145da04bf1871a, State: Running, Role: LEADER
I20260808 15:45:29.922358 8138 consensus_queue.cc:237] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } }
I20260808 15:45:29.927210 7615 catalog_manager.cc:5495] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3). New cstate: current_term: 1 leader_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90daf8b5d9bf4d5e83145da04bf1871a" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 } health_report { overall_health: UNKNOWN } } }
I20260808 15:45:29.944692 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.939566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.23.130.3:32991)
0808 15:45:29.939779 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.939787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.939805 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:45:29.939888 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.939891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.939970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.940129 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:45:29.940488 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.941195 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.942631 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.943029 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.943164 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.943630 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:45:29.944124 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:45:29.944283 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:29.944340 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 15:45:29.944429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:45:29.952791 7771 raft_consensus.cc:1261] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Refusing update from remote peer 90daf8b5d9bf4d5e83145da04bf1871a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:29.952790 7915 raft_consensus.cc:1261] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Refusing update from remote peer 90daf8b5d9bf4d5e83145da04bf1871a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:29.953778 8142 consensus_queue.cc:1035] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d021ce4625a49b0b83ac724b3924072" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:29.954149 8138 consensus_queue.cc:1035] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6dcf77b19a245b5b778c89824945586" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:29.964151 8146 mvcc.cc:205] Tried to move back new op lower bound from 7316291297076244480 to 7316291296967192576. Current Snapshot: MvccSnapshot[applied={T|T < 7316291297076244480}]
I20260808 15:45:29.980821 7616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:29.981128 7616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:29.985080 7616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Sending DeleteTablet for 3 replicas of tablet b7719c04877c478abf3cadb591e97969
I20260808 15:45:29.986187 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet b7719c04877c478abf3cadb591e97969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:29 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:36198
I20260808 15:45:29.986392 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet b7719c04877c478abf3cadb591e97969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:29 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:49304
I20260808 15:45:29.986392 7751 tablet_service.cc:1514] Processing DeleteTablet for tablet b7719c04877c478abf3cadb591e97969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:29 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:52388
I20260808 15:45:29.987373 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7658
I20260808 15:45:29.987411 8153 tablet_replica.cc:330] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: stopping tablet replica
I20260808 15:45:29.987767 8153 raft_consensus.cc:2229] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:45:29.989393 8153 raft_consensus.cc:2258] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:29.993091 8153 ts_tablet_manager.cc:1905] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:29.995275 8154 tablet_replica.cc:330] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: stopping tablet replica
I20260808 15:45:29.995801 8155 tablet_replica.cc:330] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: stopping tablet replica
I20260808 15:45:29.996326 8155 raft_consensus.cc:2229] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:29.996834 8154 raft_consensus.cc:2229] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:29.997393 8155 raft_consensus.cc:2258] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:29.997865 8154 raft_consensus.cc:2258] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:30.001003 8154 ts_tablet_manager.cc:1905] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:30.001950 8153 ts_tablet_manager.cc:1918] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:30.002202 8153 log.cc:1196] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/b7719c04877c478abf3cadb591e97969
I20260808 15:45:30.002229 8155 ts_tablet_manager.cc:1905] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:30.002948 8153 ts_tablet_manager.cc:1939] T b7719c04877c478abf3cadb591e97969 P 90daf8b5d9bf4d5e83145da04bf1871a: Deleting consensus metadata
I20260808 15:45:30.004732 7601 catalog_manager.cc:4841] TS 90daf8b5d9bf4d5e83145da04bf1871a (127.23.130.3:32991): tablet b7719c04877c478abf3cadb591e97969 (table test-table [id=aa4d794561b54b468df26cb63573386f]) successfully deleted
I20260808 15:45:30.006593 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:29.988301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.23.130.1:45071)
0808 15:45:29.988473 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:29.988480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:29.988610 (+ 130us) server_negotiation.cc:407] Connection header received
0808 15:45:29.988807 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:29.988818 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:29.988887 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:29.989001 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:29.991012 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.992020 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.993979 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:29.994465 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:29.994592 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:29.998253 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:29.998269 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:29.998287 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:29.998333 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:30.002852 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.003774 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.003783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.003787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.003961 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.004439 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.004445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.004448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.004891 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:30.005186 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:30.005857 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 15:45:30.006086 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:45:30.007488 7816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:30.011570 8154 ts_tablet_manager.cc:1918] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:30.011754 8154 log.cc:1196] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/b7719c04877c478abf3cadb591e97969
I20260808 15:45:30.012420 8154 ts_tablet_manager.cc:1939] T b7719c04877c478abf3cadb591e97969 P b6dcf77b19a245b5b778c89824945586: Deleting consensus metadata
I20260808 15:45:30.014256 7602 catalog_manager.cc:4841] TS b6dcf77b19a245b5b778c89824945586 (127.23.130.2:33853): tablet b7719c04877c478abf3cadb591e97969 (table test-table [id=aa4d794561b54b468df26cb63573386f]) successfully deleted
I20260808 15:45:30.107815 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7658
W20260808 15:45:30.129395 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:45071 (error 108)
I20260808 15:45:30.129739 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7826
I20260808 15:45:30.131812 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.131209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52410)
0808 15:45:30.131481 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.131603 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260808 15:45:30.132277 7602 catalog_manager.cc:4584] TS 6d021ce4625a49b0b83ac724b3924072 (127.23.130.1:45071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7719c04877c478abf3cadb591e97969: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
I20260808 15:45:30.146917 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.131528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.23.130.2:33853)
0808 15:45:30.131728 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:30.131734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:30.131762 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:45:30.131836 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:30.131843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:30.131933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:30.132056 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:30.133852 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:30.134845 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:30.136188 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:30.136608 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:30.136745 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:30.139601 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:30.139625 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:30.139639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:30.139676 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:30.142871 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.143563 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.143570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.143573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.143745 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.144259 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.144263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.144266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.144760 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:30.145084 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:30.146297 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0808 15:45:30.146558 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:45:30.147688 7960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:30.186218 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.185627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52414)
0808 15:45:30.185899 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.186040 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 15:45:30.187553 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.187097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52418)
0808 15:45:30.187283 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.187383 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:45:30.243156 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7826
I20260808 15:45:30.247959 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.247534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52434)
0808 15:45:30.247737 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.247817 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:45:30.249024 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.248660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52442)
0808 15:45:30.248811 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.248901 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 15:45:30.268678 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7970
I20260808 15:45:30.283896 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.269643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.23.130.3:32991)
0808 15:45:30.269852 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:30.269858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:30.269897 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:45:30.270005 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:30.270010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:30.270084 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:30.270196 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:30.271924 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:30.272753 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:30.274157 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:30.274663 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:30.274816 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:30.277376 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:30.277390 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:30.277404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:30.277435 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:30.280795 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.281547 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.281565 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.281568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.281736 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:30.282130 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:30.282136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:30.282140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:30.282584 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:30.282955 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:30.283236 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:45:30.283468 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:45:30.284641 8109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:30.329862 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.329309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52456)
0808 15:45:30.329559 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.329674 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:45:30.331179 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.330674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52468)
0808 15:45:30.330917 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.331008 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 15:45:30.407447 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7970
I20260808 15:45:30.430064 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 7580
I20260808 15:45:30.431330 7651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:30.547559 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:30.546927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:45071 (local address 127.0.0.1:52484)
0808 15:45:30.547199 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:30.547324 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:45:30.547880 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 7580
I20260808 15:45:30.574257 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--webserver_interface=127.23.130.62
--webserver_port=40319
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:41991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:30.915346 8162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:30.915656 8162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:30.915747 8162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:30.923066 8162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:30.923182 8162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:30.923228 8162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:30.923293 8162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:30.923337 8162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:30.933485 8162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:41991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:41991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=40319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:30.934864 8162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:30.936142 8162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:30.948140 8167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:30.948230 8170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:30.948171 8168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:30.949370 8162 server_base.cc:1034] running on GCE node
I20260808 15:45:30.950168 8162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:30.951707 8162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:30.952955 8162 hybrid_clock.cc:648] HybridClock initialized: now 1786203930952913 us; error 57 us; skew 500 ppm
I20260808 15:45:30.953315 8162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:30 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203930, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:30.956633 8162 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:30.958074 8162 webserver.cc:460] Webserver started at http://127.23.130.62:40319/ using document root <none> and password file <none>
I20260808 15:45:30.958802 8162 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:30.958947 8162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:30.964442 8162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:45:30.969290 8178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:30.970965 8162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:45:30.971176 8162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "b8c5a7613d44459198cd4935c67984a4"
format_stamp: "Formatted at 2026-08-08 15:45:24 on dist-test-slave-gp6n"
server_key: "16c453475c8e9e499ba96c013c15cd59"
server_key_iv: "c1d6b95fd59858a9f4dd47c6a668cffb"
server_key_version: "encryptionkey@0"
I20260808 15:45:30.972374 8162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:31.001210 8162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:31.006210 8162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:31.006573 8162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:31.027712 8162 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:41991
I20260808 15:45:31.027733 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:41991 every 8 connection(s)
I20260808 15:45:31.029369 8162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:31.030893 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8162
I20260808 15:45:31.031982 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:45071
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=42265
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:31.040053 8236 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:45:31.044955 8236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap starting.
I20260808 15:45:31.066814 8236 log.cc:824] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:31.083649 8236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:45:31.084215 8236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4: Bootstrap complete.
I20260808 15:45:31.093658 8236 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:31.095022 8236 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Initialized, Role: FOLLOWER
I20260808 15:45:31.095705 8236 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:31.095975 8236 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:31.096117 8236 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:31.096310 8236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:45:31.100171 8236 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:31.100680 8236 leader_election.cc:302] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c5a7613d44459198cd4935c67984a4; no voters:
I20260808 15:45:31.101397 8236 leader_election.cc:288] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:45:31.101547 8241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:45:31.102676 8241 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [term 3 LEADER]: Becoming Leader. State: Replica: b8c5a7613d44459198cd4935c67984a4, State: Running, Role: LEADER
I20260808 15:45:31.103309 8241 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } }
I20260808 15:45:31.103703 8236 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:31.106135 8243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c5a7613d44459198cd4935c67984a4. Latest consensus state: current_term: 3 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:31.106524 8243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:31.106626 8242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b8c5a7613d44459198cd4935c67984a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c5a7613d44459198cd4935c67984a4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 41991 } } }
I20260808 15:45:31.106933 8242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c5a7613d44459198cd4935c67984a4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:31.118460 8247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:31.127521 8247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:31.129128 8247 catalog_manager.cc:1259] Loaded cluster ID: 93d36f4984694822b871be7dd019848a
I20260808 15:45:31.129321 8247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 15:45:31.131630 8247 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbee49689ad google::LogMessage::Fail() at ??:0
@ 0x7fbee496b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbee49684de google::LogMessage::Flush() at ??:0
@ 0x7fbee4969a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbefa39b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbefa398976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbefa420610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbefa420439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260808 15:45:31.177639 8262 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fbee75b4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbee75b791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbee75b7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbee7592eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbeea86c6db start_thread at ??:0
@ 0x7fbee380261f clone at ??:0
W20260808 15:45:31.423231 8239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:31.423575 8239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:31.423731 8239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:31.431324 8239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:31.431500 8239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:31.431699 8239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:31.441758 8239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:45071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=42265
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:31.443207 8239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:31.444618 8239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:31.458313 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:31.458463 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:31.458467 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:31.459278 8239 server_base.cc:1034] running on GCE node
I20260808 15:45:31.459965 8239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:31.461014 8239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:31.462288 8239 hybrid_clock.cc:648] HybridClock initialized: now 1786203931462199 us; error 65 us; skew 500 ppm
I20260808 15:45:31.462798 8239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:31 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:31.466461 8239 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:31.467886 8239 webserver.cc:460] Webserver started at http://127.23.130.1:42265/ using document root <none> and password file <none>
I20260808 15:45:31.468590 8239 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:31.468742 8239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:31.474191 8239 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:45:31.477190 8296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:31.478569 8239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:45:31.478732 8239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "6d021ce4625a49b0b83ac724b3924072"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "d7057108be65a35b256c4c273ff7775a"
server_key_iv: "71313d4bfc8f044eaf054acd0632635b"
server_key_version: "encryptionkey@0"
I20260808 15:45:31.479678 8239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:31.500162 8239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:31.504192 8239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:31.504467 8239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:31.505489 8239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:31.507436 8303 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260808 15:45:31.512640 8303 ts_tablet_manager.cc:1852] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 15:45:31.513643 8303 ts_tablet_manager.cc:1905] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:31.520732 8303 ts_tablet_manager.cc:1918] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:31.520951 8303 log.cc:1196] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/b7719c04877c478abf3cadb591e97969
I20260808 15:45:31.521447 8303 ts_tablet_manager.cc:1939] T b7719c04877c478abf3cadb591e97969 P 6d021ce4625a49b0b83ac724b3924072: Deleting consensus metadata
I20260808 15:45:31.521847 8239 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 15:45:31.522004 8239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260808 15:45:31.522154 8239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:31.522254 8239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:31.558460 8239 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:45071
I20260808 15:45:31.558622 8423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:45071 every 8 connection(s)
I20260808 15:45:31.559816 8239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:31.564504 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8239
I20260808 15:45:31.565328 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:33853
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=46791
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:31.565357 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:31.563032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:44869)
0808 15:45:31.563809 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:31.564149 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":287,"threads_started":1}
W20260808 15:45:31.566021 8424 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:41991 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:41991: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
I20260808 15:45:31.566982 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:31.566616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:40615)
0808 15:45:31.566788 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:31.566873 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:45:31.567994 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:31.567638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:39607)
0808 15:45:31.567775 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:31.567869 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:45:31.568303 8424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:45:31.911670 8427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:31.912012 8427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:31.912216 8427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:31.920318 8427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:31.920459 8427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:31.920642 8427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:31.930994 8427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=46791
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:31.932376 8427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:31.933728 8427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:31.947397 8432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:31.947449 8434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:31.947357 8436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:31.948437 8427 server_base.cc:1034] running on GCE node
I20260808 15:45:31.949110 8427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:31.950068 8427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:31.951340 8427 hybrid_clock.cc:648] HybridClock initialized: now 1786203931951282 us; error 40 us; skew 500 ppm
I20260808 15:45:31.951694 8427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:31 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:31.955197 8427 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:31.956952 8427 webserver.cc:460] Webserver started at http://127.23.130.2:46791/ using document root <none> and password file <none>
I20260808 15:45:31.957666 8427 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:31.957780 8427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:31.963526 8427 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:45:31.966466 8444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:31.967868 8427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:45:31.968003 8427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "b6dcf77b19a245b5b778c89824945586"
format_stamp: "Formatted at 2026-08-08 15:45:25 on dist-test-slave-gp6n"
server_key: "f8dca9e09a41f10daee8f2c621961a2b"
server_key_iv: "3f9e96ccabf98cea0ba5ba45f622b0e1"
server_key_version: "encryptionkey@0"
I20260808 15:45:31.968954 8427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:31.994211 8427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:31.998693 8427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:31.998934 8427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:31.999920 8427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:32.001219 8427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:32.001307 8427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:32.001380 8427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:32.001436 8427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:32.036815 8427 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:33853
I20260808 15:45:32.036904 8563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:33853 every 8 connection(s)
I20260808 15:45:32.038295 8427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:32.043457 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.041429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:44395)
0808 15:45:32.042177 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.042552 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":281,"threads_started":1}
W20260808 15:45:32.044160 8564 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:41991 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:41991: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
I20260808 15:45:32.045027 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8427
I20260808 15:45:32.045056 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.044729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:43735)
0808 15:45:32.044882 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.044953 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:45:32.045674 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:32991
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:41991
--webserver_port=45529
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:45:32.045984 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.045606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:39313)
0808 15:45:32.045742 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.045851 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:45:32.046211 8564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:45:32.392870 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:32.393164 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:32.393353 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:32.400748 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:32.400869 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:32.401057 8568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:32.411005 8568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:32991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=45529
--tserver_master_addrs=127.23.130.62:41991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:32.412369 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:32.413719 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:32.426872 8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:32.427003 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:32.427412 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:32.431951 8568 server_base.cc:1034] running on GCE node
I20260808 15:45:32.432513 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:32.433416 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:32.434646 8568 hybrid_clock.cc:648] HybridClock initialized: now 1786203932434609 us; error 49 us; skew 500 ppm
I20260808 15:45:32.434963 8568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:32 dist-test-slave-gp6n krb5kdc[6977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203932, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:32.438197 8568 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:32.439738 8568 webserver.cc:460] Webserver started at http://127.23.130.3:45529/ using document root <none> and password file <none>
I20260808 15:45:32.440447 8568 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:32.440595 8568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:32.445981 8568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:45:32.448839 8585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:32.450336 8568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:45:32.450537 8568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "90daf8b5d9bf4d5e83145da04bf1871a"
format_stamp: "Formatted at 2026-08-08 15:45:26 on dist-test-slave-gp6n"
server_key: "aab5abebadb5df42b8801cb4b0fd17da"
server_key_iv: "2ed07d15b9fde091bc7259870ad2393a"
server_key_version: "encryptionkey@0"
I20260808 15:45:32.451535 8568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:32.466022 8568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:32.470549 8568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:32.470840 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:32.471805 8568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:32.473222 8568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:32.473348 8568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:32.473481 8568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:32.473568 8568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:32.509434 8568 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:32991
I20260808 15:45:32.509531 8705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:32991 every 8 connection(s)
I20260808 15:45:32.510982 8568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:32.515849 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.513992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:35495)
0808 15:45:32.514772 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.514995 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":280,"threads_started":1}
W20260808 15:45:32.516505 8706 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:41991 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:41991: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
I20260808 15:45:32.516883 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8568
I20260808 15:45:32.517385 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.517056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:46117)
0808 15:45:32.517202 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.517276 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:45:32.518313 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.517981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:59231)
0808 15:45:32.518119 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.518205 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:45:32.518605 8706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:45:32.569641 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:32.568839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:58175)
0808 15:45:32.569312 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:32.569406 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260808 15:45:33.047766 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:33.046898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:50741)
0808 15:45:33.047400 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:33.047517 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":229,"threads_started":1}
I20260808 15:45:33.519870 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:33.519099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:36855)
0808 15:45:33.519531 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:33.519649 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":208,"threads_started":1}
I20260808 15:45:33.571285 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:33.570634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:37261)
0808 15:45:33.571048 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:33.571131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260808 15:45:34.049528 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:34.048729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:35505)
0808 15:45:34.049220 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:34.049320 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":251,"threads_started":1}
I20260808 15:45:34.521553 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:34.520772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:43813)
0808 15:45:34.521283 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:34.521378 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":279,"threads_started":1}
I20260808 15:45:34.572963 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:34.572170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:45429)
0808 15:45:34.572632 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:34.572758 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":237,"threads_started":1}
I20260808 15:45:35.051290 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:35.050481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:41981)
0808 15:45:35.051011 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:35.051105 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
I20260808 15:45:35.523181 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:35.522513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:60801)
0808 15:45:35.522940 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:35.523019 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260808 15:45:35.574605 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:35.573822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:53955)
0808 15:45:35.574269 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:35.574351 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260808 15:45:36.053220 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:36.052294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:45997)
0808 15:45:36.052888 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:36.053002 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":351,"threads_started":1}
I20260808 15:45:36.524839 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:36.524089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:49221)
0808 15:45:36.524549 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:36.524628 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260808 15:45:36.576292 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:36.575521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:35535)
0808 15:45:36.576013 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:36.576109 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":237,"threads_started":1}
I20260808 15:45:37.055492 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:37.054533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:39185)
0808 15:45:37.055212 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:37.055298 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":422,"threads_started":1}
I20260808 15:45:37.526945 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:37.525905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:33369)
0808 15:45:37.526653 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:37.526753 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":271,"threads_started":1}
I20260808 15:45:37.577966 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:37.577235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:49709)
0808 15:45:37.577705 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:37.577788 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260808 15:45:38.057276 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:38.056550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:48173)
0808 15:45:38.057013 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:38.057113 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260808 15:45:38.528720 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:38.527969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:51383)
0808 15:45:38.528463 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:38.528552 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260808 15:45:38.579747 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:38.578956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:47985)
0808 15:45:38.579464 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:38.579542 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
I20260808 15:45:39.059439 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:39.058385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:34203)
0808 15:45:39.059024 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:39.059167 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":254,"threads_started":1}
I20260808 15:45:39.530742 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:39.529784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:57497)
0808 15:45:39.530300 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:39.530490 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":229,"threads_started":1}
I20260808 15:45:39.581358 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:39.580638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:59617)
0808 15:45:39.581074 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:39.581155 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260808 15:45:40.061257 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:40.060454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:49013)
0808 15:45:40.061004 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:40.061093 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":281,"threads_started":1}
I20260808 15:45:40.532578 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:40.531809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:36977)
0808 15:45:40.532305 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:40.532401 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260808 15:45:40.583339 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:40.582335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:35305)
0808 15:45:40.582960 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:40.583143 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":361,"threads_started":1}
I20260808 15:45:41.063112 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:41.062285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:59703)
0808 15:45:41.062839 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:41.062938 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":308,"threads_started":1}
I20260808 15:45:41.534305 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:41.533538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:38457)
0808 15:45:41.534035 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:41.534126 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":238,"threads_started":1}
I20260808 15:45:41.585040 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:41.584269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:35091)
0808 15:45:41.584778 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:41.584858 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":297,"threads_started":1}
I20260808 15:45:42.064846 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:42.064094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:47767)
0808 15:45:42.064580 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:42.064668 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260808 15:45:42.535984 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:42.535257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:49883)
0808 15:45:42.535715 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:42.535799 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260808 15:45:42.586732 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:42.585902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:49919)
0808 15:45:42.586360 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:42.586529 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
I20260808 15:45:43.066607 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:43.065814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:50817)
0808 15:45:43.066263 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:43.066424 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260808 15:45:43.537621 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:43.536896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:53729)
0808 15:45:43.537345 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:43.537441 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260808 15:45:43.588361 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:43.587660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:37847)
0808 15:45:43.588106 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:43.588199 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
I20260808 15:45:44.068269 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:44.067551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:56699)
0808 15:45:44.068030 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:44.068113 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260808 15:45:44.539521 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:44.538730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:42119)
0808 15:45:44.539239 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:44.539343 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260808 15:45:44.590155 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:44.589315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:39635)
0808 15:45:44.589835 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:44.589952 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260808 15:45:45.070135 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:45.069266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:40827)
0808 15:45:45.069847 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:45.069945 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":293,"threads_started":1}
I20260808 15:45:45.541494 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:45.540629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:57679)
0808 15:45:45.541209 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:45.541297 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":332,"threads_started":1}
I20260808 15:45:45.591902 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:45.591122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:45661)
0808 15:45:45.591592 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:45.591694 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260808 15:45:46.072077 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:46.071237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:53029)
0808 15:45:46.071727 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:46.071820 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":282,"threads_started":1}
I20260808 15:45:46.543318 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:46.542540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:50731)
0808 15:45:46.543036 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:46.543120 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260808 15:45:46.594192 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:46.593297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.1:48725)
0808 15:45:46.593900 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:46.594010 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260808 15:45:47.074167 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:47.073290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.2:37541)
0808 15:45:47.073734 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:47.073882 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":251,"threads_started":1}
I20260808 15:45:47.528724 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 8239
I20260808 15:45:47.545239 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:47.544426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:41991 (local address 127.23.130.3:60367)
0808 15:45:47.544950 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:47.545049 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260808 15:45:47.549079 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:47.529885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.23.130.1:45071)
0808 15:45:47.531843 (+ 1958us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:47.531858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:47.531896 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:45:47.533575 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:47.533586 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:47.534024 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:47.534331 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:47.535741 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.537055 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.538379 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.539216 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.539381 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:47.542173 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:47.542182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:47.542199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:47.542240 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:47.545863 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.546613 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.546621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.546628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.546826 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.547164 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.547169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.547172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.547746 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:47.547993 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:47.548348 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 15:45:47.548808 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777,"thread_start_us":1644,"threads_started":1}
I20260808 15:45:47.550625 8419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:47.635375 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 8239
I20260808 15:45:47.652761 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 8427
I20260808 15:45:47.670940 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:47.653789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.23.130.2:33853)
0808 15:45:47.654381 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:47.654441 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:47.654514 (+ 73us) server_negotiation.cc:407] Connection header received
0808 15:45:47.656084 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:47.656093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:47.656524 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:47.656875 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:47.658343 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.659716 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.660969 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.661646 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.661848 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:47.664657 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:47.664664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:47.664679 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:47.664722 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:47.667618 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.668296 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.668306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.668319 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.668499 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.668903 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.668909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.668912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.669433 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:47.669762 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:47.670204 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 15:45:47.670642 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":271,"threads_started":1}
I20260808 15:45:47.672411 8559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:47.756668 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 8427
I20260808 15:45:47.778308 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 8568
I20260808 15:45:47.797443 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:47.779402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.23.130.3:32991)
0808 15:45:47.780021 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:47.780037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:47.780078 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:45:47.781712 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:47.781721 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:47.782186 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:47.782626 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:47.784199 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.785422 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.786927 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:47.787628 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:47.787829 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:47.790863 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:47.790870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:47.790886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:47.790923 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:47.793869 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.794637 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.794646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.794653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.794831 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:47.795245 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:47.795249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:47.795251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:47.795781 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:47.796245 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:47.796696 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:45:47.797128 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":293,"threads_started":1}
I20260808 15:45:47.798993 8701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:47.882539 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 8568
2026-08-08T15:45:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25133 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:45:48 dist-test-slave-gp6n krb5kdc[8771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:45:48 dist-test-slave-gp6n krb5kdc[8771](info): set up 2 sockets
Aug 08 15:45:48 dist-test-slave-gp6n krb5kdc[8771](info): commencing operation
krb5kdc: starting...
W20260808 15:45:49.065603 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:45:49 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:45:49.359388 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:49.705920 8786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:49.706295 8786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:49.706517 8786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:49.714895 8786 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:45:49.715030 8786 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:45:49.715127 8786 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:45:49.715231 8786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:45:49.715344 8786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:49.715471 8786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:49.715571 8786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:45:49.715655 8786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:45:49.725811 8786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:49.727173 8786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:49.728518 8786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:49.741073 8791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:49.741166 8792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:49.741119 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:49.741606 8786 server_base.cc:1034] running on GCE node
I20260808 15:45:49.742600 8786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:49.744036 8786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:49.745292 8786 hybrid_clock.cc:648] HybridClock initialized: now 1786203949745224 us; error 47 us; skew 500 ppm
I20260808 15:45:49.745628 8786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:49 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:49.748968 8786 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:45:49.750550 8786 webserver.cc:460] Webserver started at http://127.23.130.62:34799/ using document root <none> and password file <none>
I20260808 15:45:49.751235 8786 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:49.751323 8786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:49.751585 8786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:49.754657 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "2b536219afa148dfaf2251a528313af4"
format_stamp: "Formatted at 2026-08-08 15:45:49 on dist-test-slave-gp6n"
server_key: "a805a1e4e4d628cd5abaf27947dbf9ff"
server_key_iv: "e05b77e89ae5f1649ba6c5c79bee8e28"
server_key_version: "encryptionkey@0"
I20260808 15:45:49.755424 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "2b536219afa148dfaf2251a528313af4"
format_stamp: "Formatted at 2026-08-08 15:45:49 on dist-test-slave-gp6n"
server_key: "a805a1e4e4d628cd5abaf27947dbf9ff"
server_key_iv: "e05b77e89ae5f1649ba6c5c79bee8e28"
server_key_version: "encryptionkey@0"
I20260808 15:45:49.760727 8786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:45:49.764305 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:49.765712 8786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:45:49.765897 8786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "2b536219afa148dfaf2251a528313af4"
format_stamp: "Formatted at 2026-08-08 15:45:49 on dist-test-slave-gp6n"
server_key: "a805a1e4e4d628cd5abaf27947dbf9ff"
server_key_iv: "e05b77e89ae5f1649ba6c5c79bee8e28"
server_key_version: "encryptionkey@0"
I20260808 15:45:49.766134 8786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:49.781363 8786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:49.785771 8786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:49.786087 8786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:49.805999 8786 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:43001
I20260808 15:45:49.806021 8856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:43001 every 8 connection(s)
I20260808 15:45:49.807442 8786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:45:49.807950 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8786
I20260808 15:45:49.808202 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:45:49.809125 24072 external_mini_cluster.cc:1428] Setting key 822f8bcecefc02e77090d8536df1d3d5
I20260808 15:45:49.812268 8857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:49.822664 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap starting.
Aug 08 15:45:49 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:49.825881 8857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:49.827085 8857 log.cc:824] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:49.829790 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: No bootstrap required, opened a new log
I20260808 15:45:49.831494 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:49.811326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.23.130.62:43001)
0808 15:45:49.812122 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:49.812134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:49.812173 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:45:49.813821 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:49.813843 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:49.814195 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:49.814658 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:49.816219 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:49.817574 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:49.818984 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:49.819658 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:49.819881 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:49.824266 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:49.824274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:49.824287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:49.824321 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:49.827565 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:49.828267 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:49.828273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:49.828279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:49.828452 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:49.828845 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:49.828849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:49.828851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:49.829343 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:49.829669 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:49.830100 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 15:45:49.830564 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":392,"threads_started":1}
I20260808 15:45:49.837951 8857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:45:49.838331 8857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:49.838477 8857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Initialized, Role: FOLLOWER
I20260808 15:45:49.839087 8857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:45:49.839337 8857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:45:49.839485 8857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:45:49.839692 8857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:49.842252 8857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:45:49.842814 8857 leader_election.cc:302] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b536219afa148dfaf2251a528313af4; no voters:
I20260808 15:45:49.843391 8857 leader_election.cc:288] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:45:49.843528 8863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:49.844225 8863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Running, Role: LEADER
I20260808 15:45:49.844841 8863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:45:49.845592 8857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:45:49.847783 8864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:45:49.848160 8864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:49.849038 8865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b536219afa148dfaf2251a528313af4. Latest consensus state: current_term: 1 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:45:49.849407 8865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:45:49.851162 8869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:45:49.857366 8869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:45:49.865622 8869 catalog_manager.cc:1347] Generated new cluster ID: f9d46d58882544deb4522c8a0e49df97
I20260808 15:45:49.865732 8869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:45:49.881287 8869 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:45:49.882901 8869 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:45:49.892720 8869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Generated new TSK 0
I20260808 15:45:49.893603 8869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:45:49.987596 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:50.364095 8884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:50.364375 8884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:50.364472 8884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:50.372351 8884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:50.372514 8884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:50.372706 8884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:45:50.382342 8884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:50.383572 8884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:50.384860 8884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:50.398293 8889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:50.398451 8892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:50.398471 8890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:50.399268 8884 server_base.cc:1034] running on GCE node
I20260808 15:45:50.399878 8884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:50.400801 8884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:50.402030 8884 hybrid_clock.cc:648] HybridClock initialized: now 1786203950401971 us; error 57 us; skew 500 ppm
I20260808 15:45:50.402374 8884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:50 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203950, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:50.405592 8884 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:45:50.407133 8884 webserver.cc:460] Webserver started at http://127.23.130.1:36985/ using document root <none> and password file <none>
I20260808 15:45:50.407770 8884 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:50.407876 8884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:50.408136 8884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:50.411046 8884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "fe6e7b7aa4c9d6605d2403d4c52f8a3f"
server_key_iv: "6719dfe2c20e588b4cf0ff3432f37311"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.411803 8884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "fe6e7b7aa4c9d6605d2403d4c52f8a3f"
server_key_iv: "6719dfe2c20e588b4cf0ff3432f37311"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.416926 8884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:45:50.420344 8900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:50.421761 8884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:45:50.421962 8884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "fe6e7b7aa4c9d6605d2403d4c52f8a3f"
server_key_iv: "6719dfe2c20e588b4cf0ff3432f37311"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.422178 8884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:50.460552 8884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:50.465337 8884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:50.465654 8884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:50.466711 8884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:50.468122 8884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:50.468202 8884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:50.468303 8884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:50.468353 8884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:50.506464 8884 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:44635
I20260808 15:45:50.506548 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:44635 every 8 connection(s)
I20260808 15:45:50.507975 8884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:45:50.514026 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 8884
I20260808 15:45:50.514277 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:45:50.515213 24072 external_mini_cluster.cc:1428] Setting key d44451508ee3fc4a770e29feef05a015
Aug 08 15:45:50 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203950, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:50.530679 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:50.510919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:53995 (local address 127.23.130.62:43001)
0808 15:45:50.511445 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:50.511453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:50.512630 (+ 1177us) server_negotiation.cc:407] Connection header received
0808 15:45:50.514788 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:50.514797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:50.514901 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:50.515062 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:50.516978 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:50.517971 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:50.519828 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:50.520240 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:50.520384 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:50.524240 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:50.524245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:50.524248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:50.524280 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:50.527436 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:50.528154 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:50.528160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:50.528164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:50.528290 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:50.528720 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:50.528724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:50.528728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:50.529095 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:50.529257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:50.530071 (+ 814us) server_negotiation.cc:299] Negotiation successful
0808 15:45:50.530323 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260808 15:45:50.531610 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:50.511637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:53995)
0808 15:45:50.512430 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:50.512494 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:50.514493 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:50.515411 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:50.515427 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:50.516151 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:50.516783 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:50.516804 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:50.518151 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:50.518161 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:50.519659 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:50.519672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:50.520501 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:50.520510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:50.520742 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:50.521483 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:50.521512 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:50.524021 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:50.527573 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:50.527581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:50.527598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:50.528024 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:50.528401 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:50.528406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:50.528409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:50.528598 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:50.529241 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:50.529248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:50.529629 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 15:45:50.529896 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 15:45:50.530324 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":302,"threads_started":1}
I20260808 15:45:50.533218 9021 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:45:50.533565 9021 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:50.534392 9021 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:45:50.536898 8821 ts_manager.cc:194] Registered new tserver with Master: e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635)
I20260808 15:45:50.538524 8821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:53995
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:45:50.581825 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:50.928151 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:50.928476 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:50.928653 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:50.935688 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:50.935833 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:50.936019 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:45:50.945349 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:50.946630 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:50.947942 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:50.961464 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:50.961388 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:50.961496 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:50.962273 9027 server_base.cc:1034] running on GCE node
I20260808 15:45:50.962898 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:50.963838 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:50.965121 9027 hybrid_clock.cc:648] HybridClock initialized: now 1786203950965032 us; error 61 us; skew 500 ppm
I20260808 15:45:50.965509 9027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:50 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203950, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:50.969498 9027 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:45:50.971189 9027 webserver.cc:460] Webserver started at http://127.23.130.2:37333/ using document root <none> and password file <none>
I20260808 15:45:50.971884 9027 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:50.972026 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:50.972352 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:50.975472 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "e4f0818acf80406d878a1f0d3fca198c"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "f330f37b663b8f7bca23d50eb431d7a4"
server_key_iv: "b79b90136b4a51d14b3fd82860fbb879"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.976282 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "e4f0818acf80406d878a1f0d3fca198c"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "f330f37b663b8f7bca23d50eb431d7a4"
server_key_iv: "b79b90136b4a51d14b3fd82860fbb879"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.981690 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:45:50.985481 9042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:50.986994 9027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:45:50.987181 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "e4f0818acf80406d878a1f0d3fca198c"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "f330f37b663b8f7bca23d50eb431d7a4"
server_key_iv: "b79b90136b4a51d14b3fd82860fbb879"
server_key_version: "encryptionkey@0"
I20260808 15:45:50.987401 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:51.020982 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:51.025127 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:51.025491 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:51.026713 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:51.028235 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:51.028342 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:51.028508 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:51.028584 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:51.063324 9027 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:44287
I20260808 15:45:51.063416 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:44287 every 8 connection(s)
I20260808 15:45:51.064652 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:45:51.070542 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9027
I20260808 15:45:51.070781 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:45:51.071406 24072 external_mini_cluster.cc:1428] Setting key d91ad9514c11a551e009ff249e1bfd8e
Aug 08 15:45:51 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203950, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:51.088239 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.067246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:58149 (local address 127.23.130.62:43001)
0808 15:45:51.067742 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.067749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.068700 (+ 951us) server_negotiation.cc:407] Connection header received
0808 15:45:51.070777 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.070785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.070885 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.071018 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:51.072711 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.073710 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.076279 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.076700 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.076798 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.080989 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:51.080998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:51.081006 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:51.081054 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:51.084755 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.085501 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.085509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.085515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.085678 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.086104 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.086111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.086117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.086525 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:51.086720 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.087741 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.087959 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260808 15:45:51.089778 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.067844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:58149)
0808 15:45:51.068536 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.068574 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.070380 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.071216 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.071224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.071810 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:51.072540 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.072560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.073891 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.073897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.076132 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.076151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.076940 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.076949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.077269 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.078017 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:51.078051 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:51.080741 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:51.084908 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:51.084915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:51.084927 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:51.085367 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:51.085785 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:51.085789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:51.085791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:51.085981 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:51.086686 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:51.086696 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:51.087214 (+ 518us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.087571 (+ 357us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.088212 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":281,"threads_started":1}
I20260808 15:45:51.091367 9162 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:45:51.091713 9162 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:51.092708 9162 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:45:51.094374 8821 ts_manager.cc:194] Registered new tserver with Master: e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
I20260808 15:45:51.095353 8821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:58149
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:45:51.141784 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:45:51.478760 9169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:45:51.479084 9169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:45:51.479262 9169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:45:51.486228 9169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:45:51.486385 9169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:45:51.486584 9169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:45:51.496142 9169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:45:51.497474 9169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:45:51.498814 9169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:45:51.514219 9178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:51.514168 9176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:45:51.514267 9175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:45:51.515579 9169 server_base.cc:1034] running on GCE node
I20260808 15:45:51.516239 9169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:45:51.517212 9169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:45:51.518501 9169 hybrid_clock.cc:648] HybridClock initialized: now 1786203951518472 us; error 52 us; skew 500 ppm
I20260808 15:45:51.518843 9169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:45:51 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203951, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:45:51.527294 9169 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:45:51.528740 9169 webserver.cc:460] Webserver started at http://127.23.130.3:33087/ using document root <none> and password file <none>
I20260808 15:45:51.529521 9169 fs_manager.cc:359] Metadata directory not provided
I20260808 15:45:51.529625 9169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:45:51.529870 9169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:45:51.532774 9169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
format_stamp: "Formatted at 2026-08-08 15:45:51 on dist-test-slave-gp6n"
server_key: "6f1a07f1eac3eb5328519ee32be74ef8"
server_key_iv: "219c46ff1c34fb28e785229cc2831433"
server_key_version: "encryptionkey@0"
I20260808 15:45:51.533504 9169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
format_stamp: "Formatted at 2026-08-08 15:45:51 on dist-test-slave-gp6n"
server_key: "6f1a07f1eac3eb5328519ee32be74ef8"
server_key_iv: "219c46ff1c34fb28e785229cc2831433"
server_key_version: "encryptionkey@0"
I20260808 15:45:51.538614 9169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:45:51.541661 9021 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:45:51.542268 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:51.543737 9169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:45:51.543870 9169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
format_stamp: "Formatted at 2026-08-08 15:45:51 on dist-test-slave-gp6n"
server_key: "6f1a07f1eac3eb5328519ee32be74ef8"
server_key_iv: "219c46ff1c34fb28e785229cc2831433"
server_key_version: "encryptionkey@0"
I20260808 15:45:51.544051 9169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:45:51.562915 9169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:45:51.567365 9169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:45:51.567625 9169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:45:51.568527 9169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:45:51.569875 9169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:45:51.569949 9169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:51.570055 9169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:45:51.570098 9169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:45:51.603070 9169 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:37707
I20260808 15:45:51.603107 9314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:37707 every 8 connection(s)
I20260808 15:45:51.604350 9169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:45:51.610652 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9169
I20260808 15:45:51.610860 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:45:51.611285 24072 external_mini_cluster.cc:1428] Setting key 45302ddbc0e9c179027bb4c901cd64d2
Aug 08 15:45:51 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203951, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:45:51.624477 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.606940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35733 (local address 127.23.130.62:43001)
0808 15:45:51.607441 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.607447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.608384 (+ 937us) server_negotiation.cc:407] Connection header received
0808 15:45:51.610178 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.610183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.610316 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.610466 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:51.612238 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.613410 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.615399 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.615811 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.615919 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.619348 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:51.619353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:51.619356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:51.619387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:51.621785 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.622509 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.622513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.622515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.622643 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.623069 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.623073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.623075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.623341 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:51.623468 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.624042 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.624200 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":225,"threads_started":1}
I20260808 15:45:51.625615 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.607519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:35733)
0808 15:45:51.608214 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.608255 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.609921 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.610717 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.610725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.611331 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:45:51.612058 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.612093 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.613605 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.613610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.615251 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.615263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.616009 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.616014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.616196 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.616810 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:45:51.616839 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:45:51.619148 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:45:51.621918 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:51.621924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:51.621935 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:51.622339 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:51.622743 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:45:51.622746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:45:51.622748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:45:51.622920 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:45:51.623449 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:45:51.623454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:45:51.623740 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.623953 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.624316 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":266,"threads_started":1}
I20260808 15:45:51.627228 9315 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:45:51.627552 9315 heartbeater.cc:464] Registering TS with master...
I20260808 15:45:51.628258 9315 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:45:51.629761 8821 ts_manager.cc:194] Registered new tserver with Master: d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707)
I20260808 15:45:51.630648 8821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:35733
I20260808 15:45:51.639696 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:45:51.657622 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.644357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.23.130.62:43001)
0808 15:45:51.644587 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.644593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.644889 (+ 296us) server_negotiation.cc:407] Connection header received
0808 15:45:51.645032 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.645035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.645109 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.645188 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:51.646724 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.647608 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.649045 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.649460 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.649558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.652140 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:51.652146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:51.652149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:51.652181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:51.655115 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.655790 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.655794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.655798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.655926 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.656250 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.656254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.656256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.656586 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:51.656789 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.657046 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.657215 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:45:51.661552 8821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:45:51.668365 8821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:45:51.718989 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.681600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.23.130.3:37707)
0808 15:45:51.705432 (+ 23832us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.706619 (+ 1187us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.707813 (+ 1194us) server_negotiation.cc:407] Connection header received
0808 15:45:51.707966 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.707977 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.708332 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.708574 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:51.709868 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.712049 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.716225 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.717680 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.717811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.718051 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.718341 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.718606 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23557,"thread_start_us":25033,"threads_started":1}
I20260808 15:45:51.718991 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.680899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37707 (local address 127.0.0.1:46720)
0808 15:45:51.707409 (+ 26510us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.707436 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.707641 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.708994 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.709005 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.709047 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:51.709599 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.709615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.712277 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.712286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.716046 (+ 3760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.716059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.717886 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.717893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.718093 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.718144 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.718272 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.718504 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26327,"thread_start_us":33904,"threads_started":1}
I20260808 15:45:51.722999 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.680247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:36840)
0808 15:45:51.681997 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.682043 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.682202 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.716195 (+ 33993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.716200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.716225 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:51.716619 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.716634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.718549 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.718554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.720748 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.720760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.722174 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.722180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.722352 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.722384 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.722502 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.722629 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"mutex_wait_us":36,"thread_start_us":1388,"threads_started":1}
I20260808 15:45:51.723057 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.680374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.23.130.1:44635)
0808 15:45:51.715623 (+ 35249us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.715632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.715663 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:45:51.715760 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.715771 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.715996 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.716211 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:51.716852 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.718354 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.720906 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.721957 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.722099 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.722326 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.722560 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.722725 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35011,"thread_start_us":34807,"threads_started":1}
I20260808 15:45:51.723470 9243 tablet_service.cc:1467] Processing CreateTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 (DEFAULT_TABLE table=test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:51.724308 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.682455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.0.0.1:57336)
0808 15:45:51.714947 (+ 32492us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.714973 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.715113 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.716502 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.716508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.716539 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:51.716994 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.717004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.719586 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.719591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.721611 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.721625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.723509 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.723516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.723675 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.723706 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.723807 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.723921 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32292,"mutex_wait_us":4,"thread_start_us":32135,"threads_started":1}
I20260808 15:45:51.725072 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f5c791dd3d484faf46d77c851df5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:51.726698 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.682788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.23.130.2:44287)
0808 15:45:51.715818 (+ 33030us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.715827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.715854 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:45:51.715952 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.715960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.716204 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.716419 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:51.717153 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.718439 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.721784 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.723236 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.723390 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.723575 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.723862 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.724041 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32867,"thread_start_us":32755,"threads_started":1}
I20260808 15:45:51.726534 8951 tablet_service.cc:1467] Processing CreateTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 (DEFAULT_TABLE table=test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:51.728046 8951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f5c791dd3d484faf46d77c851df5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:51.730300 9093 tablet_service.cc:1467] Processing CreateTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 (DEFAULT_TABLE table=test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:45:51.731642 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f5c791dd3d484faf46d77c851df5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:45:51.737164 9336 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Bootstrap starting.
I20260808 15:45:51.740468 9336 tablet_bootstrap.cc:654] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:51.741664 9336 log.cc:824] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:51.746927 9338 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Bootstrap starting.
I20260808 15:45:51.748471 9336 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: No bootstrap required, opened a new log
I20260808 15:45:51.748783 9336 ts_tablet_manager.cc:1397] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260808 15:45:51.748812 9339 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Bootstrap starting.
I20260808 15:45:51.750874 9338 tablet_bootstrap.cc:654] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:51.751518 9339 tablet_bootstrap.cc:654] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Neither blocks nor log segments found. Creating new log.
I20260808 15:45:51.752200 9338 log.cc:824] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:51.752677 9339 log.cc:824] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:45:51.755520 9338 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: No bootstrap required, opened a new log
I20260808 15:45:51.755796 9338 ts_tablet_manager.cc:1397] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:45:51.755882 9339 tablet_bootstrap.cc:492] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: No bootstrap required, opened a new log
I20260808 15:45:51.756134 9339 ts_tablet_manager.cc:1397] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 15:45:51.758113 9336 raft_consensus.cc:348] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.758518 9336 raft_consensus.cc:374] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:51.758613 9336 raft_consensus.cc:729] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61cb65b6a2c40e4acf4aa28867b8243, State: Initialized, Role: FOLLOWER
I20260808 15:45:51.759325 9336 consensus_queue.cc:260] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.761070 9315 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:45:51.761775 9336 ts_tablet_manager.cc:1428] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:45:51.766619 9338 raft_consensus.cc:348] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.766616 9339 raft_consensus.cc:348] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.767063 9338 raft_consensus.cc:374] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:51.767063 9339 raft_consensus.cc:374] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:45:51.767170 9338 raft_consensus.cc:729] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f0f9d4e1214fa0bf41b9de95ccbbc3, State: Initialized, Role: FOLLOWER
I20260808 15:45:51.767175 9339 raft_consensus.cc:729] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f0818acf80406d878a1f0d3fca198c, State: Initialized, Role: FOLLOWER
I20260808 15:45:51.767859 9339 consensus_queue.cc:260] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.768018 9338 consensus_queue.cc:260] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.768325 9344 raft_consensus.cc:482] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:45:51.768621 9344 raft_consensus.cc:504] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.769034 9162 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:45:51.769716 9339 ts_tablet_manager.cc:1428] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260808 15:45:51.770996 9344 leader_election.cc:288] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287), d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707)
I20260808 15:45:51.778079 9338 ts_tablet_manager.cc:1428] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20260808 15:45:51.785864 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.773581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:54673 (local address 127.23.130.2:44287)
0808 15:45:51.773928 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.773938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.779120 (+ 5182us) server_negotiation.cc:407] Connection header received
0808 15:45:51.779410 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.779415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.779498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.779565 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:51.780571 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.781424 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.783447 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.784725 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.784894 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.784967 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.785482 (+ 515us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.785611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260808 15:45:51.785888 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.774231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:49479 (local address 127.23.130.3:37707)
0808 15:45:51.774475 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.774484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.779221 (+ 4737us) server_negotiation.cc:407] Connection header received
0808 15:45:51.779445 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.779450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.779521 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.779582 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:45:51.780686 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.781738 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.783599 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.784801 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.784952 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.785028 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.785482 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.785611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 15:45:51.786142 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.771857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37707 (local address 127.23.130.1:49479)
0808 15:45:51.779132 (+ 7275us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.779149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.779303 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.779776 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.779781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.779832 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:51.780461 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.780473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.781920 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.781930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.783445 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.783459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.785019 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.785024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.785239 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.785349 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.785443 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.785645 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7153,"mutex_wait_us":149,"spinlock_wait_cycles":30848,"thread_start_us":6567,"threads_started":1}
I20260808 15:45:51.786391 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.771350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.23.130.1:54673)
0808 15:45:51.779006 (+ 7656us) negotiation.cc:107] Waiting for socket to connect
0808 15:45:51.779038 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:45:51.779238 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:45:51.779776 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:45:51.779781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:45:51.779832 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:45:51.780356 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.780376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.781556 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.781563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.783286 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:45:51.783299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.784991 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.785002 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:45:51.785243 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.785345 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 15:45:51.785443 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:45:51.785645 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7509,"mutex_wait_us":90,"spinlock_wait_cycles":3840,"thread_start_us":7232,"threads_started":1}
I20260808 15:45:51.787272 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f5c791dd3d484faf46d77c851df5e8" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" is_pre_election: true
I20260808 15:45:51.787428 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f5c791dd3d484faf46d77c851df5e8" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c" is_pre_election: true
I20260808 15:45:51.787781 9264 raft_consensus.cc:2454] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 0.
I20260808 15:45:51.787950 9113 raft_consensus.cc:2454] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 0.
I20260808 15:45:51.788503 8905 leader_election.cc:302] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d61cb65b6a2c40e4acf4aa28867b8243, e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters:
I20260808 15:45:51.788921 9346 raft_consensus.cc:2781] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:45:51.789067 9346 raft_consensus.cc:482] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:45:51.789213 9346 raft_consensus.cc:3037] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:51.791544 9346 raft_consensus.cc:504] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.792297 9346 leader_election.cc:288] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 election: Requested vote from peers e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287), d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707)
I20260808 15:45:51.792711 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f5c791dd3d484faf46d77c851df5e8" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c"
I20260808 15:45:51.792821 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f5c791dd3d484faf46d77c851df5e8" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
I20260808 15:45:51.792906 9113 raft_consensus.cc:3037] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:51.793074 9264 raft_consensus.cc:3037] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:45:51.795301 9113 raft_consensus.cc:2454] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 1.
I20260808 15:45:51.795622 9264 raft_consensus.cc:2454] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 1.
I20260808 15:45:51.795681 8902 leader_election.cc:302] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4f0818acf80406d878a1f0d3fca198c, e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters:
I20260808 15:45:51.795933 9346 raft_consensus.cc:2781] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:45:51.796128 9346 raft_consensus.cc:686] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 LEADER]: Becoming Leader. State: Replica: e7f0f9d4e1214fa0bf41b9de95ccbbc3, State: Running, Role: LEADER
I20260808 15:45:51.796563 9346 consensus_queue.cc:237] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } }
I20260808 15:45:51.801005 8821 catalog_manager.cc:5495] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1). New cstate: current_term: 1 leader_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } health_report { overall_health: UNKNOWN } } }
W20260808 15:45:51.816701 9163 tablet.cc:2369] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:51.846597 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.840536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.23.130.1:44635)
0808 15:45:51.840739 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.840743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.840861 (+ 118us) server_negotiation.cc:407] Connection header received
0808 15:45:51.841050 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.841054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.841131 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.841269 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:45:51.841742 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.842691 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.844245 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.844701 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.844830 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.845343 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:45:51.845983 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:45:51.846124 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.846205 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.846278 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:45:51.855401 9113 raft_consensus.cc:1261] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Refusing update from remote peer e7f0f9d4e1214fa0bf41b9de95ccbbc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:45:51.855398 9264 raft_consensus.cc:1261] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Refusing update from remote peer e7f0f9d4e1214fa0bf41b9de95ccbbc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 15:45:51.855976 9316 tablet.cc:2369] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:45:51.856527 9344 consensus_queue.cc:1035] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:51.856890 9346 consensus_queue.cc:1035] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:45:51.869634 9354 mvcc.cc:205] Tried to move back new op lower bound from 7316291386789416960 to 7316291386562801664. Current Snapshot: MvccSnapshot[applied={T|T < 7316291386789416960}]
I20260808 15:45:51.869592 9352 mvcc.cc:205] Tried to move back new op lower bound from 7316291386789416960 to 7316291386562801664. Current Snapshot: MvccSnapshot[applied={T|T < 7316291386789416960}]
I20260808 15:45:51.884124 8820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:51.884446 8820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:45:51.888263 8820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Sending DeleteTablet for 3 replicas of tablet 32f5c791dd3d484faf46d77c851df5e8
I20260808 15:45:51.889447 9243 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:51 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:46720
I20260808 15:45:51.889539 8951 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:51 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:36840
I20260808 15:45:51.889890 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 8884
I20260808 15:45:51.890621 9093 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f5c791dd3d484faf46d77c851df5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:45:51 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:57336
I20260808 15:45:51.908789 9364 tablet_replica.cc:330] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: stopping tablet replica
I20260808 15:45:51.909552 9364 raft_consensus.cc:2229] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:45:51.910116 9365 tablet_replica.cc:330] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: stopping tablet replica
I20260808 15:45:51.910622 9365 raft_consensus.cc:2229] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:51.911209 9365 raft_consensus.cc:2258] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:51.911427 9363 tablet_replica.cc:330] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: stopping tablet replica
I20260808 15:45:51.911851 9363 raft_consensus.cc:2229] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:45:51.912382 9363 raft_consensus.cc:2258] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:51.912946 9364 raft_consensus.cc:2258] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:45:51.914039 9365 ts_tablet_manager.cc:1905] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:51.916218 9364 ts_tablet_manager.cc:1905] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:45:51 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:45:51.921947 9365 ts_tablet_manager.cc:1918] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:51.922217 9365 log.cc:1196] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/32f5c791dd3d484faf46d77c851df5e8
I20260808 15:45:51.922395 9363 ts_tablet_manager.cc:1905] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:45:51.922951 9365 ts_tablet_manager.cc:1939] T 32f5c791dd3d484faf46d77c851df5e8 P e4f0818acf80406d878a1f0d3fca198c: Deleting consensus metadata
I20260808 15:45:51.924508 9364 ts_tablet_manager.cc:1918] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:51.924682 9364 log.cc:1196] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/32f5c791dd3d484faf46d77c851df5e8
I20260808 15:45:51.925182 8806 catalog_manager.cc:4841] TS e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): tablet 32f5c791dd3d484faf46d77c851df5e8 (table test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]) successfully deleted
I20260808 15:45:51.925280 9364 ts_tablet_manager.cc:1939] T 32f5c791dd3d484faf46d77c851df5e8 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Deleting consensus metadata
I20260808 15:45:51.927178 8806 catalog_manager.cc:4841] TS e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635): tablet 32f5c791dd3d484faf46d77c851df5e8 (table test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]) successfully deleted
I20260808 15:45:51.928630 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:51.891508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.23.130.1:44635)
0808 15:45:51.891729 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:51.891739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:51.900272 (+ 8533us) server_negotiation.cc:407] Connection header received
0808 15:45:51.900806 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:51.900820 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:51.900928 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:51.901017 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:51.903773 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.905420 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.910928 (+ 5508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:51.911663 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:51.911798 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:51.918089 (+ 6291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:51.918106 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:51.918130 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:51.918189 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:51.922972 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.925400 (+ 2428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.925409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.925412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.925621 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:51.926221 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:51.926226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:51.926230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:51.926727 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:51.927061 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:51.927946 (+ 885us) server_negotiation.cc:299] Negotiation successful
0808 15:45:51.928230 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:45:51.929387 9015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:51.930245 9363 ts_tablet_manager.cc:1918] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:45:51.930466 9363 log.cc:1196] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/32f5c791dd3d484faf46d77c851df5e8
I20260808 15:45:51.931046 9363 ts_tablet_manager.cc:1939] T 32f5c791dd3d484faf46d77c851df5e8 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting consensus metadata
I20260808 15:45:51.933053 8808 catalog_manager.cc:4841] TS d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): tablet 32f5c791dd3d484faf46d77c851df5e8 (table test-table [id=4cb8cfd19a8e4347ac527c75ad9c7e13]) successfully deleted
I20260808 15:45:52.032464 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 8884
I20260808 15:45:52.055699 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9027
Aug 08 15:45:52 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:45:52.072729 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:52.056529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.23.130.2:44287)
0808 15:45:52.056720 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:52.056727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:52.056787 (+ 60us) server_negotiation.cc:407] Connection header received
0808 15:45:52.057060 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:52.057065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:52.057134 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:52.057243 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:52.058800 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:52.059567 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:52.060928 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:52.061344 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:52.061498 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:52.066313 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:52.066324 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:52.066337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:52.066374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:52.069690 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:52.070648 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:52.070655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:52.070657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:52.070830 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:52.071246 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:52.071249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:52.071251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:52.071652 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:52.071883 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:52.072222 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:45:52.072403 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:45:52.073441 9155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:52.165126 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9027
I20260808 15:45:52.184996 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9169
Aug 08 15:45:52 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:45:52.201076 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:45:52.185852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.23.130.3:37707)
0808 15:45:52.186061 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:45:52.186070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:45:52.186094 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:45:52.186340 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:45:52.186352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:45:52.186481 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:45:52.186589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:45:52.187961 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:52.188713 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:52.190082 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:45:52.190601 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:45:52.190743 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:45:52.195361 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:45:52.195378 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:45:52.195394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:45:52.195431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:45:52.198055 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:52.198922 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:52.198935 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:52.198938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:52.199083 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:45:52.199489 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:45:52.199495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:45:52.199498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:45:52.199851 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:45:52.200154 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 15:45:52.200500 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 15:45:52.200728 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:45:52.201712 9309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:52.296011 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9169
I20260808 15:45:52.315727 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 8786
I20260808 15:45:52.316870 8852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:45:52.427709 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 8786
I20260808 15:46:12.453492 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--webserver_interface=127.23.130.62
--webserver_port=34799
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:12.802044 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:12.802342 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:12.802546 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:12.809825 9371 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:46:12.809926 9371 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:46:12.810032 9371 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:46:12.810128 9371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:46:12.810228 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:12.810340 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:12.810442 9371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:46:12.810521 9371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:46:12.820621 9371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:12.821979 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:12.823465 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:12.849797 9381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:12.849797 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:12.849993 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:12.851068 9371 server_base.cc:1034] running on GCE node
I20260808 15:46:12.852061 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:12.853747 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:12.855016 9371 hybrid_clock.cc:648] HybridClock initialized: now 1786203972854924 us; error 63 us; skew 500 ppm
I20260808 15:46:12.855417 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:12 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203972, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:12.864663 9371 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:46:12.866544 9371 webserver.cc:460] Webserver started at http://127.23.130.62:34799/ using document root <none> and password file <none>
I20260808 15:46:12.867349 9371 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:12.867497 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:12.873615 9371 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 15:46:12.876668 9388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:12.878080 9371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:46:12.878275 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "2b536219afa148dfaf2251a528313af4"
format_stamp: "Formatted at 2026-08-08 15:45:49 on dist-test-slave-gp6n"
server_key: "a805a1e4e4d628cd5abaf27947dbf9ff"
server_key_iv: "e05b77e89ae5f1649ba6c5c79bee8e28"
server_key_version: "encryptionkey@0"
I20260808 15:46:12.879420 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:12.898859 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:12.903211 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:12.903527 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:12.923080 9371 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:43001
I20260808 15:46:12.923117 9442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:43001 every 8 connection(s)
I20260808 15:46:12.924676 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:46:12.931885 9443 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:46:12.934264 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9371
I20260808 15:46:12.935410 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:44635
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=36985
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:46:12.935976 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap starting.
I20260808 15:46:12.960515 9443 log.cc:824] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:12.972918 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:46:12.973495 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap complete.
I20260808 15:46:12.983055 9443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:12.984028 9443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Initialized, Role: FOLLOWER
I20260808 15:46:12.984674 9443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:12.984938 9443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:46:12.985072 9443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:46:12.985288 9443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:46:12.989161 9443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:12.989832 9443 leader_election.cc:302] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b536219afa148dfaf2251a528313af4; no voters:
I20260808 15:46:12.990701 9443 leader_election.cc:288] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:46:12.990900 9448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:46:12.992125 9448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 LEADER]: Becoming Leader. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Running, Role: LEADER
I20260808 15:46:12.992918 9443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:46:12.992802 9448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:12.996547 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b536219afa148dfaf2251a528313af4. Latest consensus state: current_term: 2 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:46:12.996590 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:46:12.997069 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:12.997076 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:12.998044 9457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:46:13.009904 9457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:46:13.011471 9457 catalog_manager.cc:1259] Loaded cluster ID: f9d46d58882544deb4522c8a0e49df97
I20260808 15:46:13.011631 9457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:46:13.015698 9457 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:46:13.017212 9457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Loaded TSK: 0
I20260808 15:46:13.055827 9457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Generated new TSK 1
I20260808 15:46:13.056262 9457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:46:13.310262 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:13.310683 9445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:13.310854 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:13.319339 9445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:13.319497 9445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:13.319674 9445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:46:13.330026 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:44635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=36985
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:13.331463 9445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:13.332841 9445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:13.346429 9476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:13.346443 9475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:13.347508 9479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:13.347787 9445 server_base.cc:1034] running on GCE node
I20260808 15:46:13.348469 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:13.349602 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:13.350829 9445 hybrid_clock.cc:648] HybridClock initialized: now 1786203973350776 us; error 39 us; skew 500 ppm
I20260808 15:46:13.351184 9445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:13 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:13.355242 9445 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:46:13.357064 9445 webserver.cc:460] Webserver started at http://127.23.130.1:36985/ using document root <none> and password file <none>
I20260808 15:46:13.357992 9445 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:13.358152 9445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:13.365125 9445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 15:46:13.368546 9486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.370097 9445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:46:13.370255 9445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "fe6e7b7aa4c9d6605d2403d4c52f8a3f"
server_key_iv: "6719dfe2c20e588b4cf0ff3432f37311"
server_key_version: "encryptionkey@0"
I20260808 15:46:13.371598 9445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:13.405534 9445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:13.410065 9445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:13.410300 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:13.411356 9445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:13.412676 9445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:13.412774 9445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.412853 9445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:13.412909 9445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.446982 9445 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:44635
I20260808 15:46:13.447080 9612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:44635 every 8 connection(s)
I20260808 15:46:13.448470 9445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:46:13.458755 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9445
I20260808 15:46:13.459384 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:44287
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=37333
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:46:13 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:13.473541 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:13.451368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:42159 (local address 127.23.130.62:43001)
0808 15:46:13.451978 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:13.451990 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:13.452598 (+ 608us) server_negotiation.cc:407] Connection header received
0808 15:46:13.454810 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:13.454822 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:13.455165 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:13.455453 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:13.456840 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.458095 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.459881 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.461053 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.461262 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:13.464730 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:13.464744 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:13.464766 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:13.464816 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:13.469012 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:13.469767 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:13.469779 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:13.469789 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:13.470021 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:13.470452 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:13.470460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:13.470465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:13.471183 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:13.471385 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:13.472273 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 15:46:13.472599 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
I20260808 15:46:13.474080 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:13.451601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:42159)
0808 15:46:13.452396 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:13.452450 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:13.454233 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:13.455603 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:13.455610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:13.456115 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:13.456673 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:13.456690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.458283 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.458287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:13.459727 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:13.459752 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.461257 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.461262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:13.461448 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:13.462147 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:13.462171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:13.464430 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:13.469151 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:13.469158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:13.469169 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:13.469622 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:13.470132 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:13.470135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:13.470137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:13.470318 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:13.471315 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:13.471321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:13.471750 (+ 429us) client_negotiation.cc:769] Sending connection context
0808 15:46:13.472077 (+ 327us) client_negotiation.cc:240] Negotiation successful
0808 15:46:13.472708 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":337,"threads_started":1}
I20260808 15:46:13.476938 9613 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:46:13.477263 9613 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:13.478063 9613 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:46:13.480232 9407 ts_manager.cc:194] Registered new tserver with Master: e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635)
I20260808 15:46:13.481766 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:42159
W20260808 15:46:13.802547 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:13.802814 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:13.802960 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:13.810117 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:13.810230 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:13.810424 9617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:46:13.820159 9617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:44287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=37333
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:13.821413 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:13.822734 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:13.835687 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:13.835753 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:13.835728 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:13.836817 9617 server_base.cc:1034] running on GCE node
I20260808 15:46:13.837371 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:13.838238 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:13.839457 9617 hybrid_clock.cc:648] HybridClock initialized: now 1786203973839405 us; error 65 us; skew 500 ppm
I20260808 15:46:13.839772 9617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:13 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:13.843041 9617 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:46:13.844520 9617 webserver.cc:460] Webserver started at http://127.23.130.2:37333/ using document root <none> and password file <none>
I20260808 15:46:13.845185 9617 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:13.845342 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:13.850723 9617 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:46:13.853518 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.855029 9617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:46:13.855206 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "e4f0818acf80406d878a1f0d3fca198c"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "f330f37b663b8f7bca23d50eb431d7a4"
server_key_iv: "b79b90136b4a51d14b3fd82860fbb879"
server_key_version: "encryptionkey@0"
I20260808 15:46:13.856237 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:13.873689 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:13.877866 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:13.878139 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:13.879194 9617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:13.880501 9617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:13.880635 9617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.880754 9617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:13.880844 9617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:13.915096 9617 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:44287
I20260808 15:46:13.915199 9755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:44287 every 8 connection(s)
I20260808 15:46:13.916626 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:46:13.921953 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9617
I20260808 15:46:13.922799 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:37707
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=33087
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:46:13 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:13.937201 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:13.919171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:45575 (local address 127.23.130.62:43001)
0808 15:46:13.919358 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:13.919364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:13.920539 (+ 1175us) server_negotiation.cc:407] Connection header received
0808 15:46:13.922422 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:13.922429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:13.922521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:13.922629 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:13.924196 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.924973 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.926839 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.927311 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.927447 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:13.931048 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:13.931053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:13.931056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:13.931087 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:13.933685 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:13.934422 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:13.934429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:13.934433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:13.934555 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:13.935000 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:13.935005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:13.935008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:13.935317 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:13.935481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:13.936583 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0808 15:46:13.936841 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:46:13.938586 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:13.919684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:45575)
0808 15:46:13.920381 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:13.920419 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:13.922118 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:13.922886 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:13.922899 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:13.923472 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:13.924023 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:13.924041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.925127 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.925133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:13.926695 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:13.926707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:13.927557 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:13.927565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:13.927816 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:13.928508 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:13.928530 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:13.930801 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:13.933833 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:13.933841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:13.933858 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:13.934277 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:13.934678 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:13.934683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:13.934686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:13.934887 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:13.935642 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:13.935651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:13.936143 (+ 492us) client_negotiation.cc:769] Sending connection context
0808 15:46:13.936477 (+ 334us) client_negotiation.cc:240] Negotiation successful
0808 15:46:13.937099 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":286,"threads_started":1}
I20260808 15:46:13.940119 9756 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:46:13.940446 9756 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:13.941365 9756 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:46:13.943305 9407 ts_manager.cc:194] Registered new tserver with Master: e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
I20260808 15:46:13.944612 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:45575
W20260808 15:46:14.276399 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:14.276746 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:14.276895 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:14.285475 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:14.285600 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:14.285729 9759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:46:14.296044 9759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:37707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=33087
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:14.297657 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:14.299067 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:14.312729 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:14.312695 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:14.312775 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:14.313673 9759 server_base.cc:1034] running on GCE node
I20260808 15:46:14.314277 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:14.315454 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:14.316728 9759 hybrid_clock.cc:648] HybridClock initialized: now 1786203974316626 us; error 85 us; skew 500 ppm
I20260808 15:46:14.317097 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:14 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203974, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:14.320708 9759 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:46:14.322566 9759 webserver.cc:460] Webserver started at http://127.23.130.3:33087/ using document root <none> and password file <none>
I20260808 15:46:14.323227 9759 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:14.323350 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:14.329239 9759 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 15:46:14.332267 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:14.333729 9759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:46:14.333878 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
format_stamp: "Formatted at 2026-08-08 15:45:51 on dist-test-slave-gp6n"
server_key: "6f1a07f1eac3eb5328519ee32be74ef8"
server_key_iv: "219c46ff1c34fb28e785229cc2831433"
server_key_version: "encryptionkey@0"
I20260808 15:46:14.334990 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:14.354628 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:14.358964 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:14.359195 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:14.360216 9759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:14.361598 9759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:14.361693 9759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:14.361768 9759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:14.361828 9759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:14.396258 9759 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:37707
I20260808 15:46:14.396347 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:37707 every 8 connection(s)
I20260808 15:46:14.397779 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:46:14.407707 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9759
Aug 08 15:46:14 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203974, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:14.419160 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.400308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59595 (local address 127.23.130.62:43001)
0808 15:46:14.400481 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.400485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.401658 (+ 1173us) server_negotiation.cc:407] Connection header received
0808 15:46:14.403478 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.403483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.403572 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.403664 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:14.405066 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.405791 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.407380 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.408040 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.408188 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.412587 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:14.412593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:14.412598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:14.412634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:14.415998 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.416702 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.416707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.416711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.416834 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.417272 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.417277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.417280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.417546 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:14.417706 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.418591 (+ 885us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.418814 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:46:14.420507 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.400761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:59595)
0808 15:46:14.401465 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.401519 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.403225 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.403866 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.403873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.404365 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:14.404912 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.404928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.405925 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.405929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.407238 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.407250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.408293 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.408301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.408606 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.409663 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:14.409691 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:14.412365 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:14.416130 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:14.416139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:14.416155 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:14.416573 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:14.416947 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:14.416952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:14.416955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:14.417150 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:14.417678 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:14.417685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:14.418076 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.418484 (+ 408us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.419035 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":265,"threads_started":1}
I20260808 15:46:14.421931 9900 heartbeater.cc:347] Connected to a master server at 127.23.130.62:43001
I20260808 15:46:14.422276 9900 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:14.423040 9900 heartbeater.cc:511] Master 127.23.130.62:43001 requested a full tablet report, sending...
I20260808 15:46:14.424396 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.408718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.23.130.62:43001)
0808 15:46:14.409678 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.409682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.409696 (+ 14us) server_negotiation.cc:407] Connection header received
0808 15:46:14.409732 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.409735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.409787 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.409873 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:14.411912 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.412750 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.414324 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.415155 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.415297 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.418040 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:14.418047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:14.418051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:14.418083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:14.421463 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.422211 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.422215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.422217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.422347 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.422797 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.422803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.422806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.423115 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:14.423363 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.423742 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.423958 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":735,"threads_started":1}
I20260808 15:46:14.424546 9407 ts_manager.cc:194] Registered new tserver with Master: d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707)
I20260808 15:46:14.425577 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:59595
I20260808 15:46:14.427098 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:46:14.433220 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.427992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.23.130.62:43001)
0808 15:46:14.428205 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.428214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.428611 (+ 397us) server_negotiation.cc:407] Connection header received
0808 15:46:14.428762 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.428765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.428888 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.429003 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:46:14.429413 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.430083 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.431540 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.431943 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.432040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.432457 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:46:14.432722 (+ 265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 15:46:14.432994 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39096: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 15:46:14.433339 9904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39096: token expired
W20260808 15:46:14.433880 9323 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.130.62:43001)
I20260808 15:46:14.447463 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.434353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.23.130.62:43001)
0808 15:46:14.434544 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.434548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.434582 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:46:14.434715 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.434718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.434766 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.434839 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:14.436089 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.436758 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.438029 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.438546 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.438681 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.441492 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:14.441497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:14.441502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:14.441551 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:14.444651 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.445428 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.445432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.445435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.445561 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.445978 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.445984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.445987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.446307 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:14.446612 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.446918 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.447145 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 15:46:14.451586 9405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:46:14.459385 9405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:46:14.485316 9613 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:46:14.494637 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.471934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37707 (local address 127.0.0.1:39366)
0808 15:46:14.484084 (+ 12150us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.484134 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.484335 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.486018 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.486029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.486062 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.486680 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.486696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.489479 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.489488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.491936 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.491948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.493423 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.493429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.493622 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.493685 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.493905 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.494151 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11749,"spinlock_wait_cycles":23424,"thread_start_us":12027,"threads_started":1}
I20260808 15:46:14.494534 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.471955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.23.130.3:37707)
0808 15:46:14.483924 (+ 11969us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.483936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.484535 (+ 599us) server_negotiation.cc:407] Connection header received
0808 15:46:14.484655 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.484660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.484849 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.485061 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.486871 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.489329 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.492099 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.493178 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.493334 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.493672 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.493968 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.494144 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11777,"thread_start_us":11629,"threads_started":1}
I20260808 15:46:14.496229 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.486727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:49998)
0808 15:46:14.487390 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.487407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.487540 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.488712 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.488717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.488741 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.489239 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.489259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.491118 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.491123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.493346 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.493363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.495342 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.495348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.495567 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.495613 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.495702 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.495837 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":3,"thread_start_us":376,"threads_started":1}
I20260808 15:46:14.496528 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.487087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.23.130.1:44635)
0808 15:46:14.488097 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.488103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.488123 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:46:14.488251 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.488260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.488505 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.488711 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.489533 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.490950 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.493567 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.495111 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.495271 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.495514 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.495990 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.496179 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":730,"threads_started":1}
I20260808 15:46:14.497728 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.471940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.0.0.1:58596)
0808 15:46:14.486129 (+ 14189us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.486146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.486321 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.488885 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.488890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.488915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.489464 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.489478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.491719 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.491724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.493914 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.493929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.496948 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.496954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.497132 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.497175 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.497247 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.497376 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14088,"mutex_wait_us":83,"spinlock_wait_cycles":768,"thread_start_us":13932,"threads_started":1}
I20260808 15:46:14.498528 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.472211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.23.130.2:44287)
0808 15:46:14.485730 (+ 13519us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.485740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.486251 (+ 511us) server_negotiation.cc:407] Connection header received
0808 15:46:14.488133 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.488144 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.488649 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.489051 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.489644 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.491573 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.495245 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.496697 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.497718 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.497920 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.497990 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.498179 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13348,"thread_start_us":13235,"threads_started":1}
I20260808 15:46:14.499330 9826 tablet_service.cc:1467] Processing CreateTablet for tablet 680aef055b254d5bb51a163c293d4f66 (DEFAULT_TABLE table=test-table [id=edcc2bac01e04b7688daf1fa63d1cc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:14.500428 9539 tablet_service.cc:1467] Processing CreateTablet for tablet 680aef055b254d5bb51a163c293d4f66 (DEFAULT_TABLE table=test-table [id=edcc2bac01e04b7688daf1fa63d1cc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:14.501489 9826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680aef055b254d5bb51a163c293d4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:14.501926 9539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680aef055b254d5bb51a163c293d4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:14.501701 9685 tablet_service.cc:1467] Processing CreateTablet for tablet 680aef055b254d5bb51a163c293d4f66 (DEFAULT_TABLE table=test-table [id=edcc2bac01e04b7688daf1fa63d1cc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:14.502823 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680aef055b254d5bb51a163c293d4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:14.514894 9918 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Bootstrap starting.
I20260808 15:46:14.517347 9920 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Bootstrap starting.
I20260808 15:46:14.518184 9919 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Bootstrap starting.
I20260808 15:46:14.519940 9918 tablet_bootstrap.cc:654] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:14.521296 9918 log.cc:824] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:14.522132 9920 tablet_bootstrap.cc:654] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:14.522928 9919 tablet_bootstrap.cc:654] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:14.523428 9920 log.cc:824] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:14.524030 9919 log.cc:824] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:14.524729 9918 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: No bootstrap required, opened a new log
I20260808 15:46:14.525074 9918 ts_tablet_manager.cc:1397] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260808 15:46:14.526680 9920 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: No bootstrap required, opened a new log
I20260808 15:46:14.527055 9920 ts_tablet_manager.cc:1397] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260808 15:46:14.527253 9919 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: No bootstrap required, opened a new log
I20260808 15:46:14.527560 9919 ts_tablet_manager.cc:1397] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 15:46:14.532727 9918 raft_consensus.cc:348] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.533104 9918 raft_consensus.cc:374] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:14.533226 9918 raft_consensus.cc:729] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f0818acf80406d878a1f0d3fca198c, State: Initialized, Role: FOLLOWER
I20260808 15:46:14.533860 9918 consensus_queue.cc:260] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.534993 9756 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:46:14.536849 9918 ts_tablet_manager.cc:1428] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260808 15:46:14.538020 9919 raft_consensus.cc:348] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.538044 9920 raft_consensus.cc:348] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.538515 9920 raft_consensus.cc:374] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:14.538524 9919 raft_consensus.cc:374] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:14.538641 9919 raft_consensus.cc:729] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61cb65b6a2c40e4acf4aa28867b8243, State: Initialized, Role: FOLLOWER
I20260808 15:46:14.538630 9920 raft_consensus.cc:729] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f0f9d4e1214fa0bf41b9de95ccbbc3, State: Initialized, Role: FOLLOWER
I20260808 15:46:14.539273 9919 consensus_queue.cc:260] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.539273 9920 consensus_queue.cc:260] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.541297 9920 ts_tablet_manager.cc:1428] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 15:46:14.542270 9900 heartbeater.cc:503] Master 127.23.130.62:43001 was elected leader, sending a full tablet report...
I20260808 15:46:14.543262 9919 ts_tablet_manager.cc:1428] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260808 15:46:14.613076 9926 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:14.613263 9925 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:14.613415 9926 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.613602 9925 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.615397 9925 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
I20260808 15:46:14.615547 9926 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287), e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635)
I20260808 15:46:14.625028 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.616242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.23.130.3:53743)
0808 15:46:14.617815 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.617835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.617959 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.618581 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.618587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.618613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.618810 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.618824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.620261 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.620266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.622340 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.622355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.624186 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.624194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.624440 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.624473 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.624570 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.624718 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431}
I20260808 15:46:14.625222 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.615586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37707 (local address 127.23.130.1:46047)
0808 15:46:14.617031 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.617047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.617160 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.618348 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.618354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.618381 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.618774 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.618786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.620269 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.620274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.622568 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.622583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.624338 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.624346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.624591 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.624624 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.624707 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.624825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":140,"thread_start_us":1758,"threads_started":1}
I20260808 15:46:14.626142 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.616177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:53743 (local address 127.23.130.2:44287)
0808 15:46:14.616448 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.616457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.618080 (+ 1623us) server_negotiation.cc:407] Connection header received
0808 15:46:14.618151 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.618157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.618226 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.618303 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.618967 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.620098 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.622595 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.623854 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.624025 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.624110 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.625645 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.625794 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":77,"spinlock_wait_cycles":93568}
I20260808 15:46:14.625818 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.616047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:46047 (local address 127.23.130.3:37707)
0808 15:46:14.617936 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.617941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.617958 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:46:14.618013 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.618020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.618090 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.618168 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.618940 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.620094 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.622740 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.623977 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.624109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.624179 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.625433 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.625581 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1761}
I20260808 15:46:14.627286 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" is_pre_election: true
I20260808 15:46:14.627600 9707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c" is_pre_election: true
I20260808 15:46:14.627866 9847 raft_consensus.cc:2454] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 0.
I20260808 15:46:14.628036 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.615776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.23.130.1:36401)
0808 15:46:14.617648 (+ 1872us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.617673 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.617800 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.618983 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.618991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.619025 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.619505 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.619521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.621650 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.621659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.624765 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.624784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.627141 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.627148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.627341 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.627373 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.627473 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.627616 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"mutex_wait_us":78,"thread_start_us":891,"threads_started":1}
I20260808 15:46:14.628182 9707 raft_consensus.cc:2454] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61cb65b6a2c40e4acf4aa28867b8243 in term 0.
I20260808 15:46:14.628788 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.617273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:36401 (local address 127.23.130.2:44287)
0808 15:46:14.618046 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.618051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.618075 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:46:14.618319 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.618329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.618520 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.618646 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.619760 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.621432 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.625011 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.626697 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.626911 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.626990 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.628313 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.628454 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":618,"spinlock_wait_cycles":1408,"thread_start_us":379,"threads_started":1}
I20260808 15:46:14.628839 9777 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c; no voters:
I20260808 15:46:14.629029 9490 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d61cb65b6a2c40e4acf4aa28867b8243, e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters:
I20260808 15:46:14.629446 9926 raft_consensus.cc:2781] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:46:14.629616 9926 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:46:14.629613 9925 raft_consensus.cc:2781] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:46:14.629591 9707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c" is_pre_election: true
I20260808 15:46:14.629729 9926 raft_consensus.cc:3037] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:14.629750 9925 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:46:14.629804 9707 raft_consensus.cc:2454] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in term 0.
I20260808 15:46:14.629838 9925 raft_consensus.cc:3037] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:14.632344 9926 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.633040 9926 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [CANDIDATE]: Term 1 election: Requested vote from peers e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287), e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635)
I20260808 15:46:14.633124 9925 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.633415 9707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c"
I20260808 15:46:14.633647 9707 raft_consensus.cc:3037] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:14.634310 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
I20260808 15:46:14.634465 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.618113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:38961 (local address 127.23.130.1:44635)
0808 15:46:14.618262 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.618285 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.626377 (+ 8092us) server_negotiation.cc:407] Connection header received
0808 15:46:14.626691 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.626696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.626773 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.626839 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:14.627724 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.628807 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.632111 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.633419 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.633596 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.633675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.634057 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.634196 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:46:14.634655 9847 raft_consensus.cc:2379] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in current term 1: Already voted for candidate d61cb65b6a2c40e4acf4aa28867b8243 in this term.
I20260808 15:46:14.634488 9706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f0818acf80406d878a1f0d3fca198c"
I20260808 15:46:14.635190 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.616702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.23.130.3:38961)
0808 15:46:14.626270 (+ 9568us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.626293 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:14.626525 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:14.627022 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:14.627029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:14.627056 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:14.627487 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.627510 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.629315 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.629321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.631909 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:14.631925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.633646 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.633652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:14.633847 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.633882 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:46:14.634670 (+ 788us) client_negotiation.cc:240] Negotiation successful
0808 15:46:14.634826 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9399,"thread_start_us":9255,"threads_started":1}
I20260808 15:46:14.635550 9925 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 election: Requested vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
I20260808 15:46:14.635717 9490 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:14.636169 9925 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 15:46:14.636287 9561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" is_pre_election: true
I20260808 15:46:14.636349 9560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680aef055b254d5bb51a163c293d4f66" candidate_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
I20260808 15:46:14.636870 9561 raft_consensus.cc:2379] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d61cb65b6a2c40e4acf4aa28867b8243 in current term 1: Already voted for candidate e7f0f9d4e1214fa0bf41b9de95ccbbc3 in this term.
I20260808 15:46:14.637075 9707 raft_consensus.cc:2454] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61cb65b6a2c40e4acf4aa28867b8243 in term 1.
I20260808 15:46:14.637503 9777 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c; no voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3
I20260808 15:46:14.637791 9926 raft_consensus.cc:2781] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:46:14.638301 9926 raft_consensus.cc:686] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 LEADER]: Becoming Leader. State: Replica: d61cb65b6a2c40e4acf4aa28867b8243, State: Running, Role: LEADER
I20260808 15:46:14.638824 9926 consensus_queue.cc:237] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:14.643648 9407 catalog_manager.cc:5495] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 reported cstate change: term changed from 0 to 1, leader changed from <none> to d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } health_report { overall_health: UNKNOWN } } }
W20260808 15:46:14.649391 9901 tablet.cc:2369] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:46:14.668604 9757 tablet.cc:2369] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:46:14.690261 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.685165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.23.130.3:37707)
0808 15:46:14.685348 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.685353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.685372 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:46:14.685532 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.685536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.685634 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.685763 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:46:14.686098 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.686924 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.688187 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.688571 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.688661 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.689133 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:46:14.689644 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:46:14.689764 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.689831 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.689903 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:46:14.698553 9707 raft_consensus.cc:1261] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Refusing update from remote peer d61cb65b6a2c40e4acf4aa28867b8243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:14.698589 9561 raft_consensus.cc:1261] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Refusing update from remote peer d61cb65b6a2c40e4acf4aa28867b8243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:14.699450 9932 consensus_queue.cc:1035] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:46:14.699846 9926 consensus_queue.cc:1035] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:46:14.700939 9614 tablet.cc:2369] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:46:14.727831 9407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:14.728121 9407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:14.731528 9407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Sending DeleteTablet for 3 replicas of tablet 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:14.732904 9826 tablet_service.cc:1514] Processing DeleteTablet for tablet 680aef055b254d5bb51a163c293d4f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:39366
I20260808 15:46:14.732893 9539 tablet_service.cc:1514] Processing DeleteTablet for tablet 680aef055b254d5bb51a163c293d4f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:49998
I20260808 15:46:14.733021 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9445
I20260808 15:46:14.733964 9685 tablet_service.cc:1514] Processing DeleteTablet for tablet 680aef055b254d5bb51a163c293d4f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:14 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:58596
I20260808 15:46:14.743023 9948 tablet_replica.cc:330] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: stopping tablet replica
I20260808 15:46:14.743968 9948 raft_consensus.cc:2229] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:14.745396 9948 raft_consensus.cc:2258] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:14.749161 9947 tablet_replica.cc:330] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: stopping tablet replica
I20260808 15:46:14.749902 9947 raft_consensus.cc:2229] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:46:14.750830 9947 raft_consensus.cc:2258] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:14.752002 9946 tablet_replica.cc:330] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: stopping tablet replica
I20260808 15:46:14.752409 9946 raft_consensus.cc:2229] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:14.753882 9947 ts_tablet_manager.cc:1905] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:14.754391 9948 ts_tablet_manager.cc:1905] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:14.754873 9946 raft_consensus.cc:2258] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:14.753222 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.733851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.23.130.1:44635)
0808 15:46:14.734043 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.734049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.734337 (+ 288us) server_negotiation.cc:407] Connection header received
0808 15:46:14.734429 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.734434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.734531 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.734641 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:14.736759 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.738015 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.740135 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.740870 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.740995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.744846 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:14.744864 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:14.744883 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:14.744924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:14.749514 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.750443 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.750453 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.750457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.750634 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.751107 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.751114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.751119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.751595 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:14.751908 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.752583 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.752838 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 15:46:14.755780 9607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:14.761897 9948 ts_tablet_manager.cc:1918] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:14.762130 9948 log.cc:1196] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/680aef055b254d5bb51a163c293d4f66
I20260808 15:46:14.762843 9948 ts_tablet_manager.cc:1939] T 680aef055b254d5bb51a163c293d4f66 P e4f0818acf80406d878a1f0d3fca198c: Deleting consensus metadata
I20260808 15:46:14.763985 9947 ts_tablet_manager.cc:1918] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:14.764201 9947 log.cc:1196] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/680aef055b254d5bb51a163c293d4f66
I20260808 15:46:14.764881 9947 ts_tablet_manager.cc:1939] T 680aef055b254d5bb51a163c293d4f66 P d61cb65b6a2c40e4acf4aa28867b8243: Deleting consensus metadata
I20260808 15:46:14.764961 9390 catalog_manager.cc:4841] TS e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): tablet 680aef055b254d5bb51a163c293d4f66 (table test-table [id=edcc2bac01e04b7688daf1fa63d1cc52]) successfully deleted
I20260808 15:46:14.766968 9392 catalog_manager.cc:4841] TS d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): tablet 680aef055b254d5bb51a163c293d4f66 (table test-table [id=edcc2bac01e04b7688daf1fa63d1cc52]) successfully deleted
I20260808 15:46:14.875077 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9445
W20260808 15:46:14.902865 9390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:44635 (error 108)
I20260808 15:46:14.903021 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9617
I20260808 15:46:14.905021 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.904519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50022)
0808 15:46:14.904721 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.904844 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 15:46:14.905409 9390 catalog_manager.cc:4584] TS e7f0f9d4e1214fa0bf41b9de95ccbbc3 (127.23.130.1:44635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 680aef055b254d5bb51a163c293d4f66: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
I20260808 15:46:14.920923 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.904629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.23.130.2:44287)
0808 15:46:14.904795 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:14.904801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:14.904820 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:46:14.904896 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:14.904901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:14.904995 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:14.905116 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:14.906907 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.907635 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.909676 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:14.910093 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:14.910187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:14.913571 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:14.913584 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:14.913597 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:14.913629 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:14.917694 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.918584 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.918596 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.918599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.918759 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:14.919197 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:14.919201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:14.919204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:14.919688 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:14.920007 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:14.920336 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 15:46:14.920558 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:46:14.921593 9750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:14.959097 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.958526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50026)
0808 15:46:14.958771 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.958906 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:46:14.960330 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:14.959923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50036)
0808 15:46:14.960088 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:14.960190 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:46:15.021327 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.020771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50046)
0808 15:46:15.021027 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:15.021141 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 15:46:15.022563 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.022120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50058)
0808 15:46:15.022297 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:15.022395 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:46:15.025079 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9617
I20260808 15:46:15.047592 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9759
I20260808 15:46:15.064708 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.048634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.23.130.3:37707)
0808 15:46:15.048867 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:15.048880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:15.048904 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:46:15.048959 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:15.048962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:15.049027 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:15.049152 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:15.050496 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:15.051244 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:15.053244 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:15.053950 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:15.054060 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:15.058194 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:15.058206 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:15.058221 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:15.058265 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:15.061652 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:15.062572 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:15.062580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:15.062582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:15.062715 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:15.063168 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:15.063171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:15.063173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:15.063571 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:15.063803 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:15.064210 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 15:46:15.064381 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:46:15.065541 9891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:15.103173 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.102661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50062)
0808 15:46:15.102885 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:15.102980 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 15:46:15.104281 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.103854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50076)
0808 15:46:15.104077 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:15.104160 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 15:46:15.193851 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9759
I20260808 15:46:15.216449 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 9371
I20260808 15:46:15.217624 9438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:15.333961 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:15.333005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:44635 (local address 127.0.0.1:50092)
0808 15:46:15.333224 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:15.333349 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:46:15.334010 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 9371
I20260808 15:46:35.357563 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--webserver_interface=127.23.130.62
--webserver_port=34799
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:35.707988 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:35.708272 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:35.708384 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:35.715547 9955 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:46:35.715642 9955 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:46:35.715682 9955 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:46:35.715734 9955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:46:35.715787 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:35.715862 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:35.715907 9955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:46:35.715955 9955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:46:35.725488 9955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:35.726687 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:35.728026 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:35.740531 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:35.740603 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:35.740566 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:35.741209 9955 server_base.cc:1034] running on GCE node
I20260808 15:46:35.742209 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:35.743818 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:35.745103 9955 hybrid_clock.cc:648] HybridClock initialized: now 1786203995745041 us; error 77 us; skew 500 ppm
I20260808 15:46:35.745467 9955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:35 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203995, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:35.753656 9955 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:46:35.755239 9955 webserver.cc:460] Webserver started at http://127.23.130.62:34799/ using document root <none> and password file <none>
I20260808 15:46:35.756000 9955 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:35.756134 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:35.761737 9955 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 15:46:35.764797 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:35.766252 9955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:46:35.766510 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "2b536219afa148dfaf2251a528313af4"
format_stamp: "Formatted at 2026-08-08 15:45:49 on dist-test-slave-gp6n"
server_key: "a805a1e4e4d628cd5abaf27947dbf9ff"
server_key_iv: "e05b77e89ae5f1649ba6c5c79bee8e28"
server_key_version: "encryptionkey@0"
I20260808 15:46:35.767511 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:35.798048 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:35.802467 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:35.802773 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:35.822309 9955 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:43001
I20260808 15:46:35.822343 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:43001 every 8 connection(s)
I20260808 15:46:35.823933 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:46:35.829764 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 9955
I20260808 15:46:35.830736 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:44635
--local_ip_for_outbound_sockets=127.23.130.1
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=36985
--webserver_interface=127.23.130.1
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:46:35.831117 10029 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:46:35.834273 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap starting.
I20260808 15:46:35.859306 10029 log.cc:824] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:35.881203 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:46:35.881855 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Bootstrap complete.
I20260808 15:46:35.895361 10029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:35.896732 10029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Initialized, Role: FOLLOWER
I20260808 15:46:35.897478 10029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:35.897771 10029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:46:35.897917 10029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:46:35.898108 10029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:46:35.904062 10029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:35.904740 10029 leader_election.cc:302] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b536219afa148dfaf2251a528313af4; no voters:
I20260808 15:46:35.905515 10029 leader_election.cc:288] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:46:35.905653 10034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:46:35.907173 10034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [term 3 LEADER]: Becoming Leader. State: Replica: 2b536219afa148dfaf2251a528313af4, State: Running, Role: LEADER
I20260808 15:46:35.907748 10034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } }
I20260808 15:46:35.908049 10029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:46:35.910509 10036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b536219afa148dfaf2251a528313af4. Latest consensus state: current_term: 3 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:46:35.910535 10035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b536219afa148dfaf2251a528313af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b536219afa148dfaf2251a528313af4" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43001 } } }
I20260808 15:46:35.910992 10036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:35.911006 10035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:35.922901 10041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:46:35.931963 10041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:46:35.934299 10041 catalog_manager.cc:1259] Loaded cluster ID: f9d46d58882544deb4522c8a0e49df97
I20260808 15:46:35.934527 10041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:46:35.938833 10041 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:46:35.940611 10041 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Loaded TSK: 0
I20260808 15:46:35.940835 10041 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b536219afa148dfaf2251a528313af4: Loaded TSK: 1
F20260808 15:46:35.942092 10041 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5709dbe9ad google::LogMessage::Fail() at ??:0
I20260808 15:46:35.947242 10051 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f5709dc188a google::LogMessage::SendToLog() at ??:0
@ 0x7f5709dbe4de google::LogMessage::Flush() at ??:0
@ 0x7f5709dbfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f570db227d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f570db1173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f571f7ed1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f571f7ec783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f571f8782a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f571f8780c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f571f8c25e9 std::function<>::operator()() at ??:0
@ 0x7f571f7f0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f571f7eec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f571f876610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f571f876439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f570ca0a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f570ca0d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f570ca0d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f570c9e8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f570fcc26db start_thread at ??:0
@ 0x7f5708c5861f clone at ??:0
W20260808 15:46:36.309392 10031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:36.309700 10031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:36.309798 10031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:36.317231 10031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:36.317349 10031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:36.317466 10031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:46:36.327730 10031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:44635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=36985
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:36.329063 10031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:36.330487 10031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:36.344053 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:36.344226 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:36.344578 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:36.344862 10031 server_base.cc:1034] running on GCE node
I20260808 15:46:36.345479 10031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:36.346479 10031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:36.347707 10031 hybrid_clock.cc:648] HybridClock initialized: now 1786203996347658 us; error 57 us; skew 500 ppm
I20260808 15:46:36.348026 10031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:36 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:36.351251 10031 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:46:36.352830 10031 webserver.cc:460] Webserver started at http://127.23.130.1:36985/ using document root <none> and password file <none>
I20260808 15:46:36.353552 10031 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:36.353729 10031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:36.359344 10031 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:46:36.362269 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:36.363940 10031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:46:36.364133 10031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "fe6e7b7aa4c9d6605d2403d4c52f8a3f"
server_key_iv: "6719dfe2c20e588b4cf0ff3432f37311"
server_key_version: "encryptionkey@0"
I20260808 15:46:36.365549 10031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:36.392606 10031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:36.398787 10031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:36.399124 10031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:36.400393 10031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:36.402680 10100 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 15:46:36.412190 10031 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 15:46:36.412351 10031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.001s
I20260808 15:46:36.412526 10031 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 15:46:36.415886 10031 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 15:46:36.415997 10031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.001s
I20260808 15:46:36.416337 10100 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Bootstrap starting.
I20260808 15:46:36.424823 10100 log.cc:824] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:36.444707 10100 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:46:36.445727 10100 tablet_bootstrap.cc:492] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Bootstrap complete.
I20260808 15:46:36.446425 10100 ts_tablet_manager.cc:1397] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.001s
I20260808 15:46:36.461215 10100 raft_consensus.cc:348] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:36.463042 10031 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:44635
I20260808 15:46:36.463080 10100 raft_consensus.cc:729] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f0f9d4e1214fa0bf41b9de95ccbbc3, State: Initialized, Role: FOLLOWER
I20260808 15:46:36.463135 10216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:44635 every 8 connection(s)
I20260808 15:46:36.463944 10100 consensus_queue.cc:260] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:36.464854 10031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:46:36.466135 10100 ts_tablet_manager.cc:1428] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260808 15:46:36.468480 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10031
I20260808 15:46:36.469462 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:44287
--local_ip_for_outbound_sockets=127.23.130.2
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=37333
--webserver_interface=127.23.130.2
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:46:36.470927 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.468260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:46501)
0808 15:46:36.469290 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.469598 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":443,"threads_started":1}
W20260808 15:46:36.471711 10219 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:43001 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:43001: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
I20260808 15:46:36.472721 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.472264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:40675)
0808 15:46:36.472455 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.472561 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 15:46:36.473778 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.473306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:57563)
0808 15:46:36.473496 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.473601 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:46:36.474053 10219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:46:36.812903 10222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:36.813248 10222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:36.813438 10222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:36.820637 10222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:36.820798 10222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:36.820995 10222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:46:36.830785 10222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:44287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=37333
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:36.832120 10222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:36.833546 10222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:36.847208 10228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:36.847258 10232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:36.847286 10227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:36.848210 10222 server_base.cc:1034] running on GCE node
I20260808 15:46:36.848793 10222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:36.849687 10222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:36.850886 10222 hybrid_clock.cc:648] HybridClock initialized: now 1786203996850830 us; error 54 us; skew 500 ppm
I20260808 15:46:36.851219 10222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:36 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:36.854678 10222 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:46:36.856233 10222 webserver.cc:460] Webserver started at http://127.23.130.2:37333/ using document root <none> and password file <none>
I20260808 15:46:36.856901 10222 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:36.857007 10222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:36.862516 10222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:46:36.865223 10240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:36.866743 10222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:46:36.866931 10222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "e4f0818acf80406d878a1f0d3fca198c"
format_stamp: "Formatted at 2026-08-08 15:45:50 on dist-test-slave-gp6n"
server_key: "f330f37b663b8f7bca23d50eb431d7a4"
server_key_iv: "b79b90136b4a51d14b3fd82860fbb879"
server_key_version: "encryptionkey@0"
I20260808 15:46:36.867951 10222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:36.898622 10222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:36.903342 10222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:36.903658 10222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:36.904719 10222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:36.906147 10222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:36.906236 10222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:36.906435 10222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:36.906523 10222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 15:46:36.941229 10222 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:44287
I20260808 15:46:36.941335 10364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:44287 every 8 connection(s)
I20260808 15:46:36.942935 10222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:46:36.948030 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.946100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:52205)
0808 15:46:36.946921 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.947185 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
W20260808 15:46:36.948720 10365 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:43001 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:43001: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
I20260808 15:46:36.949698 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.949259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:41877)
0808 15:46:36.949453 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.949542 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 15:46:36.950140 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10222
I20260808 15:46:36.950928 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:36.950505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:32965)
0808 15:46:36.950670 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:36.950766 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 15:46:36.950984 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:37707
--local_ip_for_outbound_sockets=127.23.130.3
--tserver_master_addrs=127.23.130.62:43001
--webserver_port=33087
--webserver_interface=127.23.130.3
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:36.951215 10365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:46:37.298480 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:37.298774 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:37.298933 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:37.306203 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:37.306334 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:37.306540 10368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:46:37.316445 10368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:37707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=33087
--tserver_master_addrs=127.23.130.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:37.317850 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:37.319628 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:37.333427 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:37.333546 10375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:37.333518 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:37.334369 10368 server_base.cc:1034] running on GCE node
I20260808 15:46:37.335114 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:37.336148 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:37.337400 10368 hybrid_clock.cc:648] HybridClock initialized: now 1786203997337331 us; error 54 us; skew 500 ppm
I20260808 15:46:37.337783 10368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:37 dist-test-slave-gp6n krb5kdc[8771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203997, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:37.341650 10368 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:46:37.343271 10368 webserver.cc:460] Webserver started at http://127.23.130.3:33087/ using document root <none> and password file <none>
I20260808 15:46:37.343966 10368 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:37.344089 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:37.349884 10368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 15:46:37.352826 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:37.354321 10368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:46:37.354578 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "d61cb65b6a2c40e4acf4aa28867b8243"
format_stamp: "Formatted at 2026-08-08 15:45:51 on dist-test-slave-gp6n"
server_key: "6f1a07f1eac3eb5328519ee32be74ef8"
server_key_iv: "219c46ff1c34fb28e785229cc2831433"
server_key_version: "encryptionkey@0"
I20260808 15:46:37.355654 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:37.372921 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:37.377393 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:37.377676 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:37.378702 10368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:37.380035 10368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:37.380160 10368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:37.380298 10368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:37.380406 10368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:37.414891 10368 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:37707
I20260808 15:46:37.414956 10510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:37707 every 8 connection(s)
I20260808 15:46:37.416415 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:46:37.421211 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.419452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:56733)
0808 15:46:37.420152 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.420378 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":266,"threads_started":1}
W20260808 15:46:37.421857 10511 heartbeater.cc:644] Failed to heartbeat to 127.23.130.62:43001 (0 consecutive failures): Network error: Failed to ping master at 127.23.130.62:43001: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
I20260808 15:46:37.422385 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10368
I20260808 15:46:37.422755 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.422383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:39537)
0808 15:46:37.422587 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.422652 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 15:46:37.423588 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.423248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:48137)
0808 15:46:37.423385 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.423469 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:46:37.423892 10511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:46:37.475378 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.474633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:57857)
0808 15:46:37.475057 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.475142 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":200,"threads_started":1}
I20260808 15:46:37.603997 10517 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:37.604305 10517 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:37.605994 10517 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
Aug 08 15:46:37 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
Aug 08 15:46:37 dist-test-slave-gp6n krb5kdc[8771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786203996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:46:37.627964 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.606448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:44287 (local address 127.23.130.1:40861)
0808 15:46:37.608551 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.608581 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:37.608705 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:37.611186 (+ 2481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:37.611190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:37.611786 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:37.612627 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:37.612652 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.614357 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.614367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:37.615854 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:37.615866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.616886 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.616894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:37.617082 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:37.617852 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:37.617877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:37.620384 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:37.624042 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:37.624048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:37.624059 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:37.624750 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:37.625205 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:37.625208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:37.625210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:37.625406 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:37.626286 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:37.626296 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:37.626667 (+ 371us) client_negotiation.cc:769] Sending connection context
0808 15:46:37.626927 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 15:46:37.627444 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1953,"mutex_wait_us":172,"spinlock_wait_cycles":12928,"thread_start_us":1786,"threads_started":1}
I20260808 15:46:37.627954 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.606747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:40861 (local address 127.23.130.2:44287)
0808 15:46:37.608938 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:37.608953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:37.608982 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:46:37.610374 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:37.610382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:37.610847 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:37.611206 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:37.612883 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.614164 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.616005 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.616651 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.616860 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:37.620698 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:37.620711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:37.620728 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:37.620774 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:37.623861 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:37.624891 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:37.624897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:37.624902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:37.625094 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:37.625651 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:37.625654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:37.625656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:37.626144 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:37.626346 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:37.627178 (+ 832us) server_negotiation.cc:299] Negotiation successful
0808 15:46:37.627546 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2030,"thread_start_us":1911,"threads_started":1}
I20260808 15:46:37.627969 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.606560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:52185 (local address 127.23.130.3:37707)
0808 15:46:37.608230 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:37.608250 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:37.608304 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:46:37.609988 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:37.609996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:37.610369 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:37.610717 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:37.612885 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.614213 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.615995 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.616613 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.616772 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:37.620990 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:37.620997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:37.621008 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:37.621044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:37.624255 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:37.625138 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:37.625146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:37.625155 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:37.625330 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:37.625747 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:37.625750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:37.625753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:37.626205 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:37.626411 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:37.627158 (+ 747us) server_negotiation.cc:299] Negotiation successful
0808 15:46:37.627544 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1289,"threads_started":1}
I20260808 15:46:37.628042 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.606194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:37707 (local address 127.23.130.1:52185)
0808 15:46:37.606549 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.606603 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:37.607285 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:37.610896 (+ 3611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:37.610908 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:37.611782 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:37.612625 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:37.612647 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.614440 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.614452 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:37.615852 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:37.615865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:37.616812 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:37.616818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:37.617083 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:37.617923 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:37.617939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:37.620743 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:37.624421 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:37.624426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:37.624430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:37.624991 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:37.625461 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:37.625467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:37.625471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:37.625617 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:37.626342 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:37.626349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:37.626668 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 15:46:37.626927 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 15:46:37.627626 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":406,"spinlock_wait_cycles":100096}
W20260808 15:46:37.631613 10097 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
W20260808 15:46:37.632169 10095 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:37.632285 10095 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:37.632655 10517 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:46:37.952766 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:37.951879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:54439)
0808 15:46:37.952373 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:37.952519 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260808 15:46:38.425215 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:38.424416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:60999)
0808 15:46:38.424900 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:38.425016 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260808 15:46:38.477149 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:38.476409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:40641)
0808 15:46:38.476876 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:38.476975 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":235,"threads_started":1}
I20260808 15:46:38.954460 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:38.953699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:41459)
0808 15:46:38.954149 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:38.954233 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260808 15:46:39.176904 10526 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:39.177145 10526 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:39.177907 10526 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
W20260808 15:46:39.179020 10095 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
W20260808 15:46:39.179265 10097 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:39.179438 10097 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:39.179697 10526 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:46:39.426807 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:39.426077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:50433)
0808 15:46:39.426545 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:39.426628 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
I20260808 15:46:39.478942 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:39.478044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:36717)
0808 15:46:39.478619 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:39.478725 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
I20260808 15:46:39.956014 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:39.955324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:32981)
0808 15:46:39.955770 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:39.955849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260808 15:46:40.428448 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:40.427703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:56565)
0808 15:46:40.428167 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:40.428263 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260808 15:46:40.480429 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:40.479745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:59055)
0808 15:46:40.480183 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:40.480270 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260808 15:46:40.957875 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:40.956958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:42483)
0808 15:46:40.957540 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:40.957636 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":280,"threads_started":1}
I20260808 15:46:41.075258 10535 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:41.075496 10535 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:41.076179 10535 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
W20260808 15:46:41.077118 10095 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
W20260808 15:46:41.077363 10097 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:41.077521 10097 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:41.077787 10535 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:46:41.430159 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:41.429385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:36841)
0808 15:46:41.429862 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:41.429950 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260808 15:46:41.482190 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:41.481378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:41733)
0808 15:46:41.481912 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:41.481995 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":266,"threads_started":1}
I20260808 15:46:41.959630 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:41.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:47995)
0808 15:46:41.959377 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:41.959463 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260808 15:46:42.431813 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:42.431079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:49615)
0808 15:46:42.431561 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:42.431652 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":276,"threads_started":1}
I20260808 15:46:42.483770 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:42.483068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:47373)
0808 15:46:42.483514 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:42.483600 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260808 15:46:42.961189 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:42.960472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:50139)
0808 15:46:42.960926 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:42.961013 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260808 15:46:43.433610 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:43.432839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:49535)
0808 15:46:43.433336 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:43.433420 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260808 15:46:43.485369 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:43.484614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:57223)
0808 15:46:43.485089 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:43.485178 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260808 15:46:43.962942 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:43.962111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:56981)
0808 15:46:43.962666 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:43.962754 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":282,"threads_started":1}
I20260808 15:46:44.435547 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:44.434708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:41037)
0808 15:46:44.435259 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:44.435343 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":282,"threads_started":1}
I20260808 15:46:44.486985 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:44.486224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:59603)
0808 15:46:44.486705 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:44.486819 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260808 15:46:44.568120 10549 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:44.568348 10549 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:44.569123 10549 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
W20260808 15:46:44.570222 10095 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
W20260808 15:46:44.570488 10097 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:44.570637 10097 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:44.570875 10549 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:46:44.964602 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:44.963851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:55795)
0808 15:46:44.964288 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:44.964399 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260808 15:46:45.437275 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:45.436527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:57797)
0808 15:46:45.436996 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:45.437091 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260808 15:46:45.488654 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:45.487915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:51345)
0808 15:46:45.488385 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:45.488466 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":230,"threads_started":1}
I20260808 15:46:45.966308 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:45.965547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:35617)
0808 15:46:45.966071 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:45.966154 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260808 15:46:46.439558 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:46.438626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:57757)
0808 15:46:46.439271 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:46.439389 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":316,"threads_started":1}
I20260808 15:46:46.490684 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:46.489756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:51287)
0808 15:46:46.490351 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:46.490515 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":313,"threads_started":1}
I20260808 15:46:46.968039 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:46.967281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:32873)
0808 15:46:46.967747 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:46.967836 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260808 15:46:47.441547 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:47.440636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:42919)
0808 15:46:47.441216 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:47.441304 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":339,"threads_started":1}
I20260808 15:46:47.492452 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:47.491659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:35013)
0808 15:46:47.492171 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:47.492268 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
I20260808 15:46:47.969903 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:47.969110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:45413)
0808 15:46:47.969626 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:47.969733 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":256,"threads_started":1}
I20260808 15:46:47.978500 10562 raft_consensus.cc:482] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:47.978757 10562 raft_consensus.cc:504] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61cb65b6a2c40e4acf4aa28867b8243" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 37707 } } peers { permanent_uuid: "e4f0818acf80406d878a1f0d3fca198c" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 44287 } } peers { permanent_uuid: "e7f0f9d4e1214fa0bf41b9de95ccbbc3" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 44635 } }
I20260808 15:46:47.979434 10562 leader_election.cc:288] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707), e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287)
W20260808 15:46:47.980546 10097 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d61cb65b6a2c40e4acf4aa28867b8243 (127.23.130.3:37707): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
W20260808 15:46:47.980746 10095 leader_election.cc:341] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f0818acf80406d878a1f0d3fca198c (127.23.130.2:44287): Not found: Tablet not found: 680aef055b254d5bb51a163c293d4f66
I20260808 15:46:47.980957 10095 leader_election.cc:302] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f0f9d4e1214fa0bf41b9de95ccbbc3; no voters: d61cb65b6a2c40e4acf4aa28867b8243, e4f0818acf80406d878a1f0d3fca198c
I20260808 15:46:47.981230 10562 raft_consensus.cc:2726] T 680aef055b254d5bb51a163c293d4f66 P e7f0f9d4e1214fa0bf41b9de95ccbbc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:46:48.443486 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:48.442707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:57039)
0808 15:46:48.443227 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:48.443315 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":257,"threads_started":1}
I20260808 15:46:48.494158 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:48.493367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:54683)
0808 15:46:48.493893 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:48.493992 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":298,"threads_started":1}
I20260808 15:46:48.972043 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:48.971110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:52167)
0808 15:46:48.971698 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:48.971808 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":338,"threads_started":1}
I20260808 15:46:49.445273 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:49.444479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:43673)
0808 15:46:49.444970 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:49.445058 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":289,"threads_started":1}
I20260808 15:46:49.496019 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:49.495166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:39489)
0808 15:46:49.495692 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:49.495821 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260808 15:46:49.973839 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:49.972959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:45051)
0808 15:46:49.973518 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:49.973632 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260808 15:46:50.447036 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:50.446258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:43783)
0808 15:46:50.446767 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:50.446862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
I20260808 15:46:50.497982 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:50.497171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:33883)
0808 15:46:50.497684 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:50.497781 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":270,"threads_started":1}
I20260808 15:46:50.975446 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:50.974726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:53763)
0808 15:46:50.975160 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:50.975242 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260808 15:46:51.448690 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:51.447930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:32891)
0808 15:46:51.448398 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:51.448487 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260808 15:46:51.500082 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:51.499295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.1:59489)
0808 15:46:51.499778 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:51.499899 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":228,"threads_started":1}
I20260808 15:46:51.977255 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:51.976569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.2:34953)
0808 15:46:51.977002 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:51.977090 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260808 15:46:52.430969 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10031
I20260808 15:46:52.451265 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:52.450170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:43001 (local address 127.23.130.3:35575)
0808 15:46:52.450824 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:52.451001 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.62:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260808 15:46:52.453478 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:52.432028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.23.130.1:44635)
0808 15:46:52.432588 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:52.432596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:52.432627 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:46:52.432696 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:52.432703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:52.432941 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:52.433136 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:52.434891 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.436032 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.437436 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.438053 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.438216 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:52.446201 (+ 7985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:52.446215 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:52.446231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:52.446269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:52.449630 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.450803 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.450815 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.450819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.451026 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.451620 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.451624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.451627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.452049 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:52.452438 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:52.452787 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 15:46:52.453114 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":242,"threads_started":1}
I20260808 15:46:52.454886 10211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:52.548931 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10031
I20260808 15:46:52.571537 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10222
I20260808 15:46:52.586062 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:52.572442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.23.130.2:44287)
0808 15:46:52.573008 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:52.573012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:52.573030 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:46:52.573086 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:52.573090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:52.573149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:52.573244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:52.574757 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.575622 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.576878 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.577365 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.577462 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:52.580519 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:52.580524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:52.580527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:52.580571 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:52.583465 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.584178 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.584183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.584185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.584380 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.584792 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.584796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.584799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.585070 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:52.585289 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:52.585579 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 15:46:52.585755 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":337,"threads_started":1}
I20260808 15:46:52.586933 10358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:52.676575 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10222
I20260808 15:46:52.697791 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10368
I20260808 15:46:52.712239 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:52.698690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.23.130.3:37707)
0808 15:46:52.699246 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:52.699250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:52.699266 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:46:52.699320 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:52.699324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:52.699400 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:52.699523 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:52.700973 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.701790 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.703206 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:52.703666 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:52.703809 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:52.706502 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:52.706508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:52.706512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:52.706544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:52.709462 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.710171 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.710178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.710181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.710342 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:52.710738 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:52.710743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:52.710747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:52.711015 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:52.711296 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:52.711613 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 15:46:52.711840 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":267,"threads_started":1}
I20260808 15:46:52.712962 10505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:52.798120 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10368
2026-08-08T15:46:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64915 ms)
[----------] 30 tests from SecurityITest (305993 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:46:52 dist-test-slave-gp6n krb5kdc[10590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:46:52 dist-test-slave-gp6n krb5kdc[10590](info): set up 2 sockets
Aug 08 15:46:52 dist-test-slave-gp6n krb5kdc[10590](info): commencing operation
krb5kdc: starting...
W20260808 15:46:53.989885 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:46:54 dist-test-slave-gp6n krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:46:54.314736 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38695
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:54.681572 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:54.681885 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:54.682073 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:54.689628 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:46:54.689744 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:54.689834 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:54.689944 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:46:54.690033 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:46:54.700021 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:54.701411 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:54.702775 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:54.715672 10612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:54.715638 10613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:54.715732 10615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:54.716161 10606 server_base.cc:1034] running on GCE node
I20260808 15:46:54.717013 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:54.718366 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:54.719630 10606 hybrid_clock.cc:648] HybridClock initialized: now 1786204014719556 us; error 62 us; skew 500 ppm
I20260808 15:46:54.719975 10606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:54 dist-test-slave-gp6n krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:54.723097 10606 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:46:54.724541 10606 webserver.cc:460] Webserver started at http://127.23.130.62:34549/ using document root <none> and password file <none>
I20260808 15:46:54.725212 10606 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:54.725366 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:54.725679 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:54.728726 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "51ca4e8cd05f424d935d43431c2b2625"
format_stamp: "Formatted at 2026-08-08 15:46:54 on dist-test-slave-gp6n"
server_key: "e9dd9a6533da47de5bef92822b76cd34"
server_key_iv: "97f715c091dc5afb8df27a27232cc656"
server_key_version: "encryptionkey@0"
I20260808 15:46:54.729511 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "51ca4e8cd05f424d935d43431c2b2625"
format_stamp: "Formatted at 2026-08-08 15:46:54 on dist-test-slave-gp6n"
server_key: "e9dd9a6533da47de5bef92822b76cd34"
server_key_iv: "97f715c091dc5afb8df27a27232cc656"
server_key_version: "encryptionkey@0"
I20260808 15:46:54.734928 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 15:46:54.738624 10624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:54.740028 10606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:46:54.740214 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "51ca4e8cd05f424d935d43431c2b2625"
format_stamp: "Formatted at 2026-08-08 15:46:54 on dist-test-slave-gp6n"
server_key: "e9dd9a6533da47de5bef92822b76cd34"
server_key_iv: "97f715c091dc5afb8df27a27232cc656"
server_key_version: "encryptionkey@0"
I20260808 15:46:54.740422 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:54.757495 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:54.761766 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:54.762080 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:54.781805 10606 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:38695
I20260808 15:46:54.781824 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:38695 every 8 connection(s)
I20260808 15:46:54.783232 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:46:54.786809 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10606
I20260808 15:46:54.787045 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:46:54.787103 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:54.787750 24072 external_mini_cluster.cc:1428] Setting key c3f7b04f19f06df471c5b8a8015ce71e
I20260808 15:46:54.800014 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: Bootstrap starting.
Aug 08 15:46:54 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:54.803774 10683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:54.805083 10683 log.cc:824] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:54.808524 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: No bootstrap required, opened a new log
I20260808 15:46:54.811391 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:54.790053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.23.130.62:38695)
0808 15:46:54.790910 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:54.790922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:54.790960 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:46:54.792516 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:54.792536 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:54.792913 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:54.793313 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:54.794879 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:54.796144 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:54.797518 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:54.798250 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:54.798533 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:54.803269 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:54.803277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:54.803292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:54.803334 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:54.806995 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:54.807908 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:54.807916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:54.807922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:54.808078 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:54.808620 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:54.808627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:54.808631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:54.809172 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:54.809458 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:54.809915 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 15:46:54.810231 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":406,"threads_started":1}
I20260808 15:46:54.816748 10683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } }
I20260808 15:46:54.817142 10683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:54.817261 10683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ca4e8cd05f424d935d43431c2b2625, State: Initialized, Role: FOLLOWER
I20260808 15:46:54.817819 10683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } }
I20260808 15:46:54.818065 10683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:46:54.818199 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:46:54.818367 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:54.820634 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } }
I20260808 15:46:54.821149 10683 leader_election.cc:302] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ca4e8cd05f424d935d43431c2b2625; no voters:
I20260808 15:46:54.821734 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:46:54.821918 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:46:54.822844 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [term 1 LEADER]: Becoming Leader. State: Replica: 51ca4e8cd05f424d935d43431c2b2625, State: Running, Role: LEADER
I20260808 15:46:54.823482 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } }
I20260808 15:46:54.824090 10683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:46:54.826467 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51ca4e8cd05f424d935d43431c2b2625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } } }
I20260808 15:46:54.826995 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:54.826556 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ca4e8cd05f424d935d43431c2b2625. Latest consensus state: current_term: 1 leader_uuid: "51ca4e8cd05f424d935d43431c2b2625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ca4e8cd05f424d935d43431c2b2625" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38695 } } }
I20260808 15:46:54.828203 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:54.830827 10698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:46:54.836620 10698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:46:54.845821 10698 catalog_manager.cc:1347] Generated new cluster ID: a438fff5bb874dd0a2312b789648427b
I20260808 15:46:54.845959 10698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:46:54.867018 10698 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:46:54.868826 10698 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:46:54.880347 10698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: Generated new TSK 0
I20260808 15:46:54.881299 10698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:46:54.974582 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:55.370290 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:55.370638 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:55.370736 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:55.378224 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:55.378345 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:55.378535 10715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:46:55.388518 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:55.389835 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:55.391191 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:55.404627 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:55.404675 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:55.404974 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:55.405855 10715 server_base.cc:1034] running on GCE node
I20260808 15:46:55.406538 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:55.407485 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:55.408730 10715 hybrid_clock.cc:648] HybridClock initialized: now 1786204015408675 us; error 43 us; skew 500 ppm
I20260808 15:46:55.409060 10715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:55 dist-test-slave-gp6n krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:55.412310 10715 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:46:55.413831 10715 webserver.cc:460] Webserver started at http://127.23.130.1:44837/ using document root <none> and password file <none>
I20260808 15:46:55.414582 10715 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:55.414733 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:55.415055 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:55.417932 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "d5f5ddbfce4e4c7980806c0f63819aac"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "631fd6a65c5a82839e3c0716c693481e"
server_key_iv: "ae35f0ce72e9b23cbc76e110102b10d1"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.418761 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f5ddbfce4e4c7980806c0f63819aac"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "631fd6a65c5a82839e3c0716c693481e"
server_key_iv: "ae35f0ce72e9b23cbc76e110102b10d1"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.424019 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:46:55.427677 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:55.429034 10715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:46:55.429219 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "d5f5ddbfce4e4c7980806c0f63819aac"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "631fd6a65c5a82839e3c0716c693481e"
server_key_iv: "ae35f0ce72e9b23cbc76e110102b10d1"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.429448 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:55.444257 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:55.448545 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:55.448851 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:55.449836 10715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:55.451282 10715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:55.451387 10715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:55.451540 10715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:55.451620 10715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:55.487735 10715 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:34697
I20260808 15:46:55.487787 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:34697 every 8 connection(s)
I20260808 15:46:55.489111 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:46:55.494148 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10715
I20260808 15:46:55.494383 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:46:55.495129 24072 external_mini_cluster.cc:1428] Setting key 4935fc8c7670a8a9b4162d3cecb96234
Aug 08 15:46:55 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:55.512291 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:55.491764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:58949 (local address 127.23.130.62:38695)
0808 15:46:55.492229 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:55.492233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:55.493226 (+ 993us) server_negotiation.cc:407] Connection header received
0808 15:46:55.495348 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:55.495354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:55.495447 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:55.495584 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:55.497945 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:55.499106 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:55.500774 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:55.501510 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:55.501663 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:55.505268 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:55.505278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:55.505286 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:55.505335 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:55.509015 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:55.509753 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:55.509761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:55.509767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:55.509931 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:55.510459 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:55.510467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:55.510473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:55.510868 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:55.511034 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:55.511859 (+ 825us) server_negotiation.cc:299] Negotiation successful
0808 15:46:55.512039 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260808 15:46:55.513844 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:55.492346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38695 (local address 127.23.130.1:58949)
0808 15:46:55.493066 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:55.493103 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:55.495012 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:55.495912 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:55.495926 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:55.496760 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:55.497712 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:55.497736 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:55.499318 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:55.499323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:55.500615 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:55.500627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:55.501734 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:55.501738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:55.501945 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:55.502691 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:55.502714 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:55.504976 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:55.509157 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:55.509164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:55.509177 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:55.509632 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:55.510053 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:55.510059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:55.510062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:55.510312 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:55.511001 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:55.511007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:55.511422 (+ 415us) client_negotiation.cc:769] Sending connection context
0808 15:46:55.511770 (+ 348us) client_negotiation.cc:240] Negotiation successful
0808 15:46:55.512361 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":274,"threads_started":1}
I20260808 15:46:55.515538 10856 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38695
I20260808 15:46:55.515861 10856 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:55.516636 10856 heartbeater.cc:511] Master 127.23.130.62:38695 requested a full tablet report, sending...
I20260808 15:46:55.519007 10644 ts_manager.cc:194] Registered new tserver with Master: d5f5ddbfce4e4c7980806c0f63819aac (127.23.130.1:34697)
I20260808 15:46:55.520439 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:58949
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:46:55.566166 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:55.921891 10864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:55.922181 10864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:55.922360 10864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:55.929811 10864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:55.929926 10864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:55.930174 10864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:46:55.940467 10864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:55.941876 10864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:55.943248 10864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:55.956955 10870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:55.956889 10872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:55.956847 10869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:55.957858 10864 server_base.cc:1034] running on GCE node
I20260808 15:46:55.958542 10864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:55.959501 10864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:55.960708 10864 hybrid_clock.cc:648] HybridClock initialized: now 1786204015960665 us; error 35 us; skew 500 ppm
I20260808 15:46:55.961040 10864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:55 dist-test-slave-gp6n krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:55.964332 10864 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:46:55.965807 10864 webserver.cc:460] Webserver started at http://127.23.130.2:43137/ using document root <none> and password file <none>
I20260808 15:46:55.966576 10864 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:55.966713 10864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:55.967059 10864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:55.970012 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "d18f884c2c8e45f781d6a7860304fb20"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "db40939be6c2f3d0b32b21dee374047e"
server_key_iv: "7d7c309131ede1ca9994e21e35b414ca"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.970834 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "d18f884c2c8e45f781d6a7860304fb20"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "db40939be6c2f3d0b32b21dee374047e"
server_key_iv: "7d7c309131ede1ca9994e21e35b414ca"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.976179 10864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:46:55.979764 10879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:55.981058 10864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 15:46:55.981238 10864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "d18f884c2c8e45f781d6a7860304fb20"
format_stamp: "Formatted at 2026-08-08 15:46:55 on dist-test-slave-gp6n"
server_key: "db40939be6c2f3d0b32b21dee374047e"
server_key_iv: "7d7c309131ede1ca9994e21e35b414ca"
server_key_version: "encryptionkey@0"
I20260808 15:46:55.981437 10864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:56.012167 10864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:56.017648 10864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:56.017917 10864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:56.018965 10864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:56.020349 10864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:56.020424 10864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:56.020516 10864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:56.020560 10864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:56.054817 10864 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:41951
I20260808 15:46:56.054862 11003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:41951 every 8 connection(s)
I20260808 15:46:56.056183 10864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:46:56.066058 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 10864
I20260808 15:46:56.066256 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:46:56.066876 24072 external_mini_cluster.cc:1428] Setting key f16ab9b1cce8d9fa99010bf4c95e2e54
Aug 08 15:46:56 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:56.077587 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.058778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:42489 (local address 127.23.130.62:38695)
0808 15:46:56.059246 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.059252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.060241 (+ 989us) server_negotiation.cc:407] Connection header received
0808 15:46:56.062034 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.062039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.062109 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.062209 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:56.063730 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.064445 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.065990 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.066693 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.066825 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.070470 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:56.070480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:56.070487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:56.070534 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:56.074212 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.074940 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.074955 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.074961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.075129 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.075563 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.075570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.075576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.075969 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:56.076125 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.076786 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.077068 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":209,"threads_started":1}
I20260808 15:46:56.078181 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.059373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38695 (local address 127.23.130.2:42489)
0808 15:46:56.060072 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.060119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.061772 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.062463 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.062474 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.063013 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:56.063580 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.063597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.064593 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.064597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.065828 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.065840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.066902 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.066907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.067187 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.067895 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:56.067927 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:56.070216 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:56.074348 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:56.074354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:56.074367 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:56.074825 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:56.075238 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:56.075241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:56.075243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:56.075455 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:56.076086 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:56.076091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:56.076392 (+ 301us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.076651 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.077013 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":290,"threads_started":1}
I20260808 15:46:56.079751 11004 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38695
I20260808 15:46:56.080090 11004 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:56.080968 11004 heartbeater.cc:511] Master 127.23.130.62:38695 requested a full tablet report, sending...
I20260808 15:46:56.082597 10644 ts_manager.cc:194] Registered new tserver with Master: d18f884c2c8e45f781d6a7860304fb20 (127.23.130.2:41951)
I20260808 15:46:56.083415 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:42489
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:46:56.137178 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:46:56.491130 11012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:56.491536 11012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:56.491701 11012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:56.502457 11012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:56.502625 11012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:56.502831 11012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:46:56.516021 11012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:56.517414 11012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:56.518851 11012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:46:56.523458 10856 heartbeater.cc:503] Master 127.23.130.62:38695 was elected leader, sending a full tablet report...
W20260808 15:46:56.535218 11021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:56.535193 11019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:56.535336 11018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:56.536019 11012 server_base.cc:1034] running on GCE node
I20260808 15:46:56.536684 11012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:56.537621 11012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:56.538847 11012 hybrid_clock.cc:648] HybridClock initialized: now 1786204016538794 us; error 66 us; skew 500 ppm
I20260808 15:46:56.539165 11012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:46:56 dist-test-slave-gp6n krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204016, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:46:56.542384 11012 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:46:56.543928 11012 webserver.cc:460] Webserver started at http://127.23.130.3:42291/ using document root <none> and password file <none>
I20260808 15:46:56.544659 11012 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:56.544813 11012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:56.545132 11012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:56.548144 11012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "44b96a4b90a34758bd5bbe9e11b14bb3"
format_stamp: "Formatted at 2026-08-08 15:46:56 on dist-test-slave-gp6n"
server_key: "45741c3359427f77490752b2cde81ae8"
server_key_iv: "aaa8a302d6428a966d75241883dd94e1"
server_key_version: "encryptionkey@0"
I20260808 15:46:56.548929 11012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "44b96a4b90a34758bd5bbe9e11b14bb3"
format_stamp: "Formatted at 2026-08-08 15:46:56 on dist-test-slave-gp6n"
server_key: "45741c3359427f77490752b2cde81ae8"
server_key_iv: "aaa8a302d6428a966d75241883dd94e1"
server_key_version: "encryptionkey@0"
I20260808 15:46:56.554320 11012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:46:56.557789 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:56.559199 11012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:46:56.559376 11012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "44b96a4b90a34758bd5bbe9e11b14bb3"
format_stamp: "Formatted at 2026-08-08 15:46:56 on dist-test-slave-gp6n"
server_key: "45741c3359427f77490752b2cde81ae8"
server_key_iv: "aaa8a302d6428a966d75241883dd94e1"
server_key_version: "encryptionkey@0"
I20260808 15:46:56.559600 11012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:56.572029 11012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:56.576365 11012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:56.576673 11012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:56.577610 11012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:56.579130 11012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:56.579236 11012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:56.579380 11012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:56.579463 11012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:56.614221 11012 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:40269
I20260808 15:46:56.614269 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:40269 every 8 connection(s)
I20260808 15:46:56.615619 11012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:46:56.617975 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11012
I20260808 15:46:56.618196 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:46:56.618811 24072 external_mini_cluster.cc:1428] Setting key 6f5e36197368555d632d7898e7c230c2
Aug 08 15:46:56 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204016, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:46:56.637328 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.618312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:48115 (local address 127.23.130.62:38695)
0808 15:46:56.618833 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.618840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.620017 (+ 1177us) server_negotiation.cc:407] Connection header received
0808 15:46:56.621889 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.621893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.621957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.622046 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:56.623568 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.624415 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.626336 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.626783 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.626883 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.630753 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:56.630759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:56.630762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:56.630795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:56.634709 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.635387 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.635391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.635394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.635537 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.635911 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.635914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.635916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.636170 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:56.636278 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.636887 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.637048 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260808 15:46:56.638295 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.619014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38695 (local address 127.23.130.3:48115)
0808 15:46:56.619851 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.619900 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.621644 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.622248 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.622255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.622861 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:56.623416 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.623432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.624577 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.624582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.626192 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.626206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.626980 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.626985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.627188 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.627995 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:46:56.628019 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:56.630546 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:56.634835 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:56.634841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:56.634853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:56.635276 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:56.635632 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:46:56.635635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:46:56.635637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:46:56.635814 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:46:56.636272 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:56.636278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:56.636577 (+ 299us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.636789 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.637158 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":374,"threads_started":1}
I20260808 15:46:56.639643 11153 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38695
I20260808 15:46:56.639950 11153 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:56.640714 11153 heartbeater.cc:511] Master 127.23.130.62:38695 requested a full tablet report, sending...
I20260808 15:46:56.642093 10644 ts_manager.cc:194] Registered new tserver with Master: 44b96a4b90a34758bd5bbe9e11b14bb3 (127.23.130.3:40269)
I20260808 15:46:56.642979 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:48115
I20260808 15:46:56.646624 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:46:56.664271 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.651294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.23.130.62:38695)
0808 15:46:56.651460 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.651467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.651933 (+ 466us) server_negotiation.cc:407] Connection header received
0808 15:46:56.652134 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.652140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.652218 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.652324 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:56.653752 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.654563 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.656319 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.656721 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.656823 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.659359 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:56.659364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:56.659367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:56.659397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:56.661767 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.662481 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.662492 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.662495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.662615 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:56.663032 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:56.663035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:56.663037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:56.663336 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:56.663560 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.663817 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.663994 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:46:56.668191 10644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:46:56.675123 10644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:46:56.722204 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.688394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.23.130.1:34697)
0808 15:46:56.714274 (+ 25880us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.714282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.714306 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:46:56.714423 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.714430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.714657 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.714860 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:56.715604 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.717124 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.720030 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.720984 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721143 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.721391 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.721655 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.721824 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25688,"thread_start_us":25506,"threads_started":1}
I20260808 15:46:56.722302 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.687441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51006)
0808 15:46:56.712713 (+ 25272us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.712737 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.712851 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.714866 (+ 2015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.714872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.714902 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:56.715425 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.715435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.717285 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.717291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.719884 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.719894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.721182 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.721378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.721433 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.721608 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.721824 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25147,"mutex_wait_us":193,"thread_start_us":24237,"threads_started":1}
I20260808 15:46:56.722836 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.687107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.23.130.2:41951)
0808 15:46:56.713563 (+ 26456us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.713572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.713608 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:46:56.713723 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.713734 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.714054 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.714266 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:56.715551 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.716577 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.719766 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.721147 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721266 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.721428 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.722305 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.722520 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26296,"thread_start_us":26132,"threads_started":1}
I20260808 15:46:56.723253 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.686888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:41951 (local address 127.0.0.1:49486)
0808 15:46:56.712060 (+ 25172us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.712103 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.712264 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.714535 (+ 2271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.714540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.714566 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:56.715325 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.715343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.716709 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.716716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.719612 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.719627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.721654 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721667 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.721980 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.722080 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.722158 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.722900 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24919,"mutex_wait_us":882,"thread_start_us":26053,"threads_started":1}
I20260808 15:46:56.723673 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.687774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:40269 (local address 127.0.0.1:59838)
0808 15:46:56.713562 (+ 25788us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.713579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.713701 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.714446 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.714465 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.714501 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:56.715051 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.715080 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.717193 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.717200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.719423 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.719436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.721685 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.721972 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.722076 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.722158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.722710 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25682,"mutex_wait_us":314,"spinlock_wait_cycles":7424,"thread_start_us":26388,"threads_started":1}
I20260808 15:46:56.724242 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.688379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.23.130.3:40269)
0808 15:46:56.712977 (+ 24598us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.712987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.713709 (+ 722us) server_negotiation.cc:407] Connection header received
0808 15:46:56.713846 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.713858 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.714155 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.714375 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:56.715288 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.717017 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.719594 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.721093 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.721239 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.721464 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.723159 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.723283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24383,"thread_start_us":24611,"threads_started":1}
I20260808 15:46:56.726728 10785 tablet_service.cc:1467] Processing CreateTablet for tablet d46001d0af014ac281fd43a3b1fe8476 (DEFAULT_TABLE table=test-table [id=f0da323e84654afb91f19a0cd3fac9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:56.726930 10935 tablet_service.cc:1467] Processing CreateTablet for tablet d46001d0af014ac281fd43a3b1fe8476 (DEFAULT_TABLE table=test-table [id=f0da323e84654afb91f19a0cd3fac9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:56.728039 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46001d0af014ac281fd43a3b1fe8476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:56.728314 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46001d0af014ac281fd43a3b1fe8476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:56.727972 11078 tablet_service.cc:1467] Processing CreateTablet for tablet d46001d0af014ac281fd43a3b1fe8476 (DEFAULT_TABLE table=test-table [id=f0da323e84654afb91f19a0cd3fac9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:56.729261 11078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46001d0af014ac281fd43a3b1fe8476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:56.739804 11176 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Bootstrap starting.
I20260808 15:46:56.743129 11177 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Bootstrap starting.
I20260808 15:46:56.743260 11176 tablet_bootstrap.cc:654] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:56.744675 11176 log.cc:824] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:56.747165 11178 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Bootstrap starting.
I20260808 15:46:56.747278 11177 tablet_bootstrap.cc:654] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:56.748898 11177 log.cc:824] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:56.752288 11178 tablet_bootstrap.cc:654] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:56.752820 11177 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: No bootstrap required, opened a new log
I20260808 15:46:56.752928 11176 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: No bootstrap required, opened a new log
I20260808 15:46:56.753129 11177 ts_tablet_manager.cc:1397] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260808 15:46:56.753217 11176 ts_tablet_manager.cc:1397] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260808 15:46:56.753544 11178 log.cc:824] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:56.756539 11178 tablet_bootstrap.cc:492] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: No bootstrap required, opened a new log
I20260808 15:46:56.756781 11178 ts_tablet_manager.cc:1397] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:46:56.760538 11177 raft_consensus.cc:348] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.760896 11177 raft_consensus.cc:374] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:56.761037 11177 raft_consensus.cc:729] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18f884c2c8e45f781d6a7860304fb20, State: Initialized, Role: FOLLOWER
I20260808 15:46:56.761652 11177 consensus_queue.cc:260] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.762791 11004 heartbeater.cc:503] Master 127.23.130.62:38695 was elected leader, sending a full tablet report...
I20260808 15:46:56.763408 11177 ts_tablet_manager.cc:1428] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Time spent starting tablet: real 0.010s user 0.003s sys 0.008s
I20260808 15:46:56.763800 11176 raft_consensus.cc:348] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.764264 11176 raft_consensus.cc:374] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:56.764359 11176 raft_consensus.cc:729] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f5ddbfce4e4c7980806c0f63819aac, State: Initialized, Role: FOLLOWER
I20260808 15:46:56.764923 11176 consensus_queue.cc:260] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.767635 11178 raft_consensus.cc:348] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.767982 11176 ts_tablet_manager.cc:1428] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260808 15:46:56.768080 11178 raft_consensus.cc:374] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:56.768167 11178 raft_consensus.cc:729] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b96a4b90a34758bd5bbe9e11b14bb3, State: Initialized, Role: FOLLOWER
I20260808 15:46:56.768859 11178 consensus_queue.cc:260] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.770076 11153 heartbeater.cc:503] Master 127.23.130.62:38695 was elected leader, sending a full tablet report...
I20260808 15:46:56.770682 11178 ts_tablet_manager.cc:1428] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260808 15:46:56.808350 11006 tablet.cc:2369] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:46:56.867383 11154 tablet.cc:2369] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:46:56.907585 11182 raft_consensus.cc:482] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:56.907887 11182 raft_consensus.cc:504] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.909592 11182 leader_election.cc:288] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f5ddbfce4e4c7980806c0f63819aac (127.23.130.1:34697), 44b96a4b90a34758bd5bbe9e11b14bb3 (127.23.130.3:40269)
I20260808 15:46:56.917300 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.910111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:49967 (local address 127.23.130.1:34697)
0808 15:46:56.910366 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.910374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.911162 (+ 788us) server_negotiation.cc:407] Connection header received
0808 15:46:56.911247 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.911252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.911334 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.911439 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:56.912265 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.913411 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.915492 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.916298 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.916492 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.916586 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.916886 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.917029 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:46:56.917331 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.910183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:38651 (local address 127.23.130.3:40269)
0808 15:46:56.910544 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.910550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.911018 (+ 468us) server_negotiation.cc:407] Connection header received
0808 15:46:56.911252 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.911257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.911334 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.911438 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:56.912265 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.913426 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.915519 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.916295 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.916499 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.916579 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.916931 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.917060 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260808 15:46:56.917466 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.909901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:40269 (local address 127.23.130.2:38651)
0808 15:46:56.910857 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.910880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.911060 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.911481 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.911486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.911516 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:56.912102 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.912111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.913602 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.913606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.915326 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.915340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.916594 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.916603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.916748 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.916771 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.916886 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.917014 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":5,"thread_start_us":514,"threads_started":1}
I20260808 15:46:56.917690 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.910019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.23.130.2:49967)
0808 15:46:56.911008 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:56.911023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:56.911134 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:56.911481 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:56.911486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:56.911579 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:56.912094 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.912108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.913576 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.913583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.915315 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:56.915328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.916550 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.916558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:56.916714 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.916741 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:46:56.916818 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 15:46:56.916977 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":108,"spinlock_wait_cycles":23808,"thread_start_us":578,"threads_started":1}
I20260808 15:46:56.918736 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46001d0af014ac281fd43a3b1fe8476" candidate_uuid: "d18f884c2c8e45f781d6a7860304fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" is_pre_election: true
I20260808 15:46:56.918733 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46001d0af014ac281fd43a3b1fe8476" candidate_uuid: "d18f884c2c8e45f781d6a7860304fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" is_pre_election: true
I20260808 15:46:56.919236 11100 raft_consensus.cc:2454] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d18f884c2c8e45f781d6a7860304fb20 in term 0.
I20260808 15:46:56.919237 10805 raft_consensus.cc:2454] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d18f884c2c8e45f781d6a7860304fb20 in term 0.
I20260808 15:46:56.919937 10883 leader_election.cc:302] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44b96a4b90a34758bd5bbe9e11b14bb3, d18f884c2c8e45f781d6a7860304fb20, d5f5ddbfce4e4c7980806c0f63819aac; no voters:
I20260808 15:46:56.920382 11182 raft_consensus.cc:2781] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:46:56.920529 11182 raft_consensus.cc:482] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:46:56.920647 11182 raft_consensus.cc:3037] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:56.923077 11182 raft_consensus.cc:504] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.923959 11182 leader_election.cc:288] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [CANDIDATE]: Term 1 election: Requested vote from peers d5f5ddbfce4e4c7980806c0f63819aac (127.23.130.1:34697), 44b96a4b90a34758bd5bbe9e11b14bb3 (127.23.130.3:40269)
I20260808 15:46:56.924226 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46001d0af014ac281fd43a3b1fe8476" candidate_uuid: "d18f884c2c8e45f781d6a7860304fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f5ddbfce4e4c7980806c0f63819aac"
I20260808 15:46:56.924486 10805 raft_consensus.cc:3037] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:56.924515 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46001d0af014ac281fd43a3b1fe8476" candidate_uuid: "d18f884c2c8e45f781d6a7860304fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3"
I20260808 15:46:56.924726 11100 raft_consensus.cc:3037] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:56.926995 10805 raft_consensus.cc:2454] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d18f884c2c8e45f781d6a7860304fb20 in term 1.
I20260808 15:46:56.927044 11100 raft_consensus.cc:2454] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d18f884c2c8e45f781d6a7860304fb20 in term 1.
I20260808 15:46:56.927479 10883 leader_election.cc:302] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b96a4b90a34758bd5bbe9e11b14bb3, d18f884c2c8e45f781d6a7860304fb20; no voters:
I20260808 15:46:56.927788 11182 raft_consensus.cc:2781] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:46:56.928251 11182 raft_consensus.cc:686] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 1 LEADER]: Becoming Leader. State: Replica: d18f884c2c8e45f781d6a7860304fb20, State: Running, Role: LEADER
I20260808 15:46:56.928973 11182 consensus_queue.cc:237] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } }
I20260808 15:46:56.933876 10644 catalog_manager.cc:5495] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 reported cstate change: term changed from 0 to 1, leader changed from <none> to d18f884c2c8e45f781d6a7860304fb20 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "d18f884c2c8e45f781d6a7860304fb20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18f884c2c8e45f781d6a7860304fb20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 } health_report { overall_health: UNKNOWN } } }
W20260808 15:46:56.991672 10857 tablet.cc:2369] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:46:56.994058 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:56.988658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.23.130.2:41951)
0808 15:46:56.988852 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:56.988856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:56.988954 (+ 98us) server_negotiation.cc:407] Connection header received
0808 15:46:56.989131 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:56.989135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:56.989224 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:56.989371 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:46:56.989720 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.990477 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.991871 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:56.992505 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:56.992643 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:56.993042 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:46:56.993562 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:46:56.993675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:56.993776 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 15:46:56.993847 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:46:57.002686 11100 raft_consensus.cc:1261] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 1 FOLLOWER]: Refusing update from remote peer d18f884c2c8e45f781d6a7860304fb20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:57.002714 10805 raft_consensus.cc:1261] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 1 FOLLOWER]: Refusing update from remote peer d18f884c2c8e45f781d6a7860304fb20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:57.003748 11187 consensus_queue.cc:1035] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f5ddbfce4e4c7980806c0f63819aac" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:46:57.004101 11182 consensus_queue.cc:1035] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b96a4b90a34758bd5bbe9e11b14bb3" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:46:57.031936 10644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:57.032223 10644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:57.036094 10644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51ca4e8cd05f424d935d43431c2b2625: Sending DeleteTablet for 3 replicas of tablet d46001d0af014ac281fd43a3b1fe8476
I20260808 15:46:57.037287 10935 tablet_service.cc:1514] Processing DeleteTablet for tablet d46001d0af014ac281fd43a3b1fe8476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:57 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:49486
I20260808 15:46:57.037393 11078 tablet_service.cc:1514] Processing DeleteTablet for tablet d46001d0af014ac281fd43a3b1fe8476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:57 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260808 15:46:57.037818 10785 tablet_service.cc:1514] Processing DeleteTablet for tablet d46001d0af014ac281fd43a3b1fe8476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:57 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:51006
I20260808 15:46:57.039494 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10715
I20260808 15:46:57.046780 11205 tablet_replica.cc:330] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: stopping tablet replica
I20260808 15:46:57.047379 11205 raft_consensus.cc:2229] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:57.048080 11205 raft_consensus.cc:2258] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:57.048988 11203 tablet_replica.cc:330] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: stopping tablet replica
I20260808 15:46:57.051218 11205 ts_tablet_manager.cc:1905] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:57.051772 11203 raft_consensus.cc:2229] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:57.054562 11203 raft_consensus.cc:2258] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:57.060698 11205 ts_tablet_manager.cc:1918] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 08 15:46:57 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:46:57.062755 11202 tablet_replica.cc:330] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: stopping tablet replica
I20260808 15:46:57.063024 11203 ts_tablet_manager.cc:1905] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:57.069873 11202 raft_consensus.cc:2229] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:46:57.069515 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.041489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.23.130.1:34697)
0808 15:46:57.041722 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:57.041731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:57.046544 (+ 4813us) server_negotiation.cc:407] Connection header received
0808 15:46:57.047641 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:57.047649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:57.047764 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:57.047868 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:57.050484 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.052012 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.054603 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.055570 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.055699 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:57.063444 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:57.063456 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:57.063469 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:57.063505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:57.066597 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.067331 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.067340 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.067344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.067500 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.067898 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.067903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.067906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.068284 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:57.068585 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:57.068906 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 15:46:57.069138 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:46:57.070827 11205 log.cc:1196] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/d46001d0af014ac281fd43a3b1fe8476
I20260808 15:46:57.071126 11202 raft_consensus.cc:2258] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:57.071156 10851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:57.073061 11203 ts_tablet_manager.cc:1918] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:57.073254 11203 log.cc:1196] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/d46001d0af014ac281fd43a3b1fe8476
I20260808 15:46:57.073504 11202 ts_tablet_manager.cc:1905] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:57.073899 11203 ts_tablet_manager.cc:1939] T d46001d0af014ac281fd43a3b1fe8476 P 44b96a4b90a34758bd5bbe9e11b14bb3: Deleting consensus metadata
I20260808 15:46:57.076352 10627 catalog_manager.cc:4841] TS 44b96a4b90a34758bd5bbe9e11b14bb3 (127.23.130.3:40269): tablet d46001d0af014ac281fd43a3b1fe8476 (table test-table [id=f0da323e84654afb91f19a0cd3fac9f1]) successfully deleted
I20260808 15:46:57.080729 11202 ts_tablet_manager.cc:1918] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:57.080896 11202 log.cc:1196] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/d46001d0af014ac281fd43a3b1fe8476
I20260808 15:46:57.081485 11202 ts_tablet_manager.cc:1939] T d46001d0af014ac281fd43a3b1fe8476 P d18f884c2c8e45f781d6a7860304fb20: Deleting consensus metadata
I20260808 15:46:57.083227 10626 catalog_manager.cc:4841] TS d18f884c2c8e45f781d6a7860304fb20 (127.23.130.2:41951): tablet d46001d0af014ac281fd43a3b1fe8476 (table test-table [id=f0da323e84654afb91f19a0cd3fac9f1]) successfully deleted
I20260808 15:46:57.175770 11205 ts_tablet_manager.cc:1939] T d46001d0af014ac281fd43a3b1fe8476 P d5f5ddbfce4e4c7980806c0f63819aac: Deleting consensus metadata
I20260808 15:46:57.176571 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10715
I20260808 15:46:57.197458 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10864
W20260808 15:46:57.197433 10630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:34697 (error 108)
I20260808 15:46:57.200709 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.200206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51028)
0808 15:46:57.200398 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.200526 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 15:46:57.200943 10630 catalog_manager.cc:4584] TS d5f5ddbfce4e4c7980806c0f63819aac (127.23.130.1:34697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d46001d0af014ac281fd43a3b1fe8476: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Aug 08 15:46:57 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:46:57.217293 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.198667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.23.130.2:41951)
0808 15:46:57.198853 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:57.198859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:57.198878 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:46:57.198962 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:57.198968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:57.199058 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:57.199177 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:57.201027 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.202304 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.203674 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.204373 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.204503 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:57.208906 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:57.208923 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:57.208944 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:57.208991 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:57.213041 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.214109 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.214122 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.214126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.214369 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.214949 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.214956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.214960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.215589 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:57.215946 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:57.216606 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 15:46:57.216892 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:46:57.218120 10998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:57.254854 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.254252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51030)
0808 15:46:57.254556 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.254670 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:46:57.256000 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.255595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51034)
0808 15:46:57.255754 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.255852 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:46:57.316581 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.315980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51038)
0808 15:46:57.316234 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.316349 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 15:46:57.317757 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.317340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51040)
0808 15:46:57.317533 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.317621 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 15:46:57.338167 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10864
I20260808 15:46:57.361034 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11012
Aug 08 15:46:57 dist-test-slave-gp6n krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:46:57.377283 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.361810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.23.130.3:40269)
0808 15:46:57.362018 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:57.362024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:57.362135 (+ 111us) server_negotiation.cc:407] Connection header received
0808 15:46:57.362372 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:57.362376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:57.362777 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:57.362908 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:57.364416 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.365147 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.366478 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:57.366882 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:57.367026 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:57.371563 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:57.371579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:57.371594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:46:57.371634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:57.374362 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.375149 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.375157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.375159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.375302 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:46:57.375710 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:46:57.375716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:46:57.375719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:46:57.376057 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:57.376362 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:57.376685 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 15:46:57.376909 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:46:57.377938 11146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:57.398626 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.397998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51054)
0808 15:46:57.398231 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.398342 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 15:46:57.399740 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.399355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51062)
0808 15:46:57.399568 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.399640 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 15:46:57.468017 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11012
I20260808 15:46:57.487859 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 10606
I20260808 15:46:57.489142 10677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:57.603374 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:57.602705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:34697 (local address 127.0.0.1:51076)
0808 15:46:57.602921 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:57.603010 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:46:57.603993 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 10606
2026-08-08T15:46:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4812 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T15:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:46:57Z Disabled control of system clock
I20260808 15:46:57.697533 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37455
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:46:58.040545 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:58.040866 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:58.040990 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:58.048427 11215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:46:58.048544 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:58.048583 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:58.048635 11215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:46:58.048681 11215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:46:58.058321 11215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:58.059614 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:58.061446 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:58.073836 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:58.073727 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:58.073727 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:58.075210 11215 server_base.cc:1034] running on GCE node
I20260808 15:46:58.076006 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:58.077456 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:58.078706 11215 hybrid_clock.cc:648] HybridClock initialized: now 1786204018078655 us; error 59 us; skew 500 ppm
I20260808 15:46:58.079102 11215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:46:58.087378 11215 webserver.cc:460] Webserver started at http://127.23.130.62:32925/ using document root <none> and password file <none>
I20260808 15:46:58.088109 11215 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:58.088208 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:58.088516 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:58.091761 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "210c07d9baf84228bbb78525ad651f52"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "86d4e421a77bf92d0472692e8914ebcc"
server_key_iv: "86c401cf7f2d6fdd70d8e7b917ac2833"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.092615 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "210c07d9baf84228bbb78525ad651f52"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "86d4e421a77bf92d0472692e8914ebcc"
server_key_iv: "86c401cf7f2d6fdd70d8e7b917ac2833"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.098057 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:46:58.101850 11229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:58.103397 11215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:46:58.103590 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "210c07d9baf84228bbb78525ad651f52"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "86d4e421a77bf92d0472692e8914ebcc"
server_key_iv: "86c401cf7f2d6fdd70d8e7b917ac2833"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.103850 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:58.127027 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:58.127969 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:58.128304 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:58.149030 11215 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37455
I20260808 15:46:58.149051 11289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37455 every 8 connection(s)
I20260808 15:46:58.150604 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:46:58.153285 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11215
I20260808 15:46:58.153522 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:46:58.154058 24072 external_mini_cluster.cc:1428] Setting key acfece0b8d51d3072e584304a33ec1e6
I20260808 15:46:58.155141 11290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:58.171149 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: Bootstrap starting.
I20260808 15:46:58.171424 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:58.156266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.23.130.62:37455)
0808 15:46:58.157343 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:58.157358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:58.157412 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:46:58.160079 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:58.160109 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:58.160762 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:58.161242 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:58.161826 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.163993 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.165634 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.166792 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.167000 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:58.168365 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:58.168380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:58.168406 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:58.169158 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:58.169257 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:58.169298 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:58.169405 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:58.169672 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 15:46:58.170021 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":347,"threads_started":1}
I20260808 15:46:58.174547 11290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:58.175554 11290 log.cc:824] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:58.178282 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: No bootstrap required, opened a new log
I20260808 15:46:58.185611 11290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } }
I20260808 15:46:58.186043 11290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:58.186160 11290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210c07d9baf84228bbb78525ad651f52, State: Initialized, Role: FOLLOWER
I20260808 15:46:58.186767 11290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } }
I20260808 15:46:58.187017 11290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:46:58.187165 11290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:46:58.187340 11290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:58.189487 11290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } }
I20260808 15:46:58.189980 11290 leader_election.cc:302] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210c07d9baf84228bbb78525ad651f52; no voters:
I20260808 15:46:58.190740 11290 leader_election.cc:288] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:46:58.190981 11295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:46:58.192140 11295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [term 1 LEADER]: Becoming Leader. State: Replica: 210c07d9baf84228bbb78525ad651f52, State: Running, Role: LEADER
I20260808 15:46:58.192662 11295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } }
I20260808 15:46:58.193014 11290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:46:58.195919 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210c07d9baf84228bbb78525ad651f52. Latest consensus state: current_term: 1 leader_uuid: "210c07d9baf84228bbb78525ad651f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } } }
I20260808 15:46:58.195930 11296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210c07d9baf84228bbb78525ad651f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210c07d9baf84228bbb78525ad651f52" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37455 } } }
I20260808 15:46:58.196482 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:58.196489 11296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52 [sys.catalog]: This master's current role is: LEADER
I20260808 15:46:58.198340 11301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:46:58.204253 11301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:46:58.214330 11301 catalog_manager.cc:1347] Generated new cluster ID: e78d3845a74b4a929708dd2e8c7ef470
I20260808 15:46:58.214483 11301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:46:58.254654 11301 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:46:58.256031 11301 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:46:58.265853 11301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: Generated new TSK 0
I20260808 15:46:58.266769 11301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:46:58.279930 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--builtin_ntp_servers=127.23.130.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:46:58.635064 11317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:58.635353 11317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:58.635493 11317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:58.642620 11317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:58.642719 11317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:58.642925 11317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:46:58.651889 11317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:58.653219 11317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:58.655025 11317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:58.668244 11322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:58.668337 11325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:58.668319 11323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:58.669258 11317 server_base.cc:1034] running on GCE node
I20260808 15:46:58.669929 11317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:58.670996 11317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:58.672235 11317 hybrid_clock.cc:648] HybridClock initialized: now 1786204018672201 us; error 50 us; skew 500 ppm
I20260808 15:46:58.672564 11317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:46:58.675016 11317 webserver.cc:460] Webserver started at http://127.23.130.1:39227/ using document root <none> and password file <none>
I20260808 15:46:58.675724 11317 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:58.675860 11317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:58.676226 11317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:58.679563 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "a5cd8b3bba69437988bc3fe25f89d7b0"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "48f6416af8508297e1a208a8c7280946"
server_key_iv: "b05be53477d8d3759813cf53a764cf32"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.680364 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "a5cd8b3bba69437988bc3fe25f89d7b0"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "48f6416af8508297e1a208a8c7280946"
server_key_iv: "b05be53477d8d3759813cf53a764cf32"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.685492 11317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:46:58.688933 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:58.690222 11317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:46:58.690457 11317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "a5cd8b3bba69437988bc3fe25f89d7b0"
format_stamp: "Formatted at 2026-08-08 15:46:58 on dist-test-slave-gp6n"
server_key: "48f6416af8508297e1a208a8c7280946"
server_key_iv: "b05be53477d8d3759813cf53a764cf32"
server_key_version: "encryptionkey@0"
I20260808 15:46:58.690690 11317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:58.727421 11317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:58.728353 11317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:58.728688 11317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:58.729745 11317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:58.731395 11317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:58.731524 11317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:58.731662 11317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:58.731752 11317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:58.766599 11317 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:40723
I20260808 15:46:58.766649 11454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:40723 every 8 connection(s)
I20260808 15:46:58.767961 11317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:46:58.778105 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11317
I20260808 15:46:58.778299 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:46:58.778894 24072 external_mini_cluster.cc:1428] Setting key 62dc6b40d27aa8bdcb882282ed02236c
I20260808 15:46:58.780931 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:58.770757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:54735 (local address 127.23.130.62:37455)
0808 15:46:58.771198 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:58.771205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:58.772526 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 15:46:58.774273 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:58.774277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:58.774343 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:58.774505 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:58.775538 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.776225 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.777759 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.778435 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.778572 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:58.780020 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:58.780026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:58.780030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:58.780052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:58.780123 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:58.780139 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:58.780235 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:58.780470 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 15:46:58.780555 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260808 15:46:58.782126 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:58.771455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37455 (local address 127.23.130.1:54735)
0808 15:46:58.772347 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:58.772386 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:58.774024 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:58.774698 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:58.774705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:58.774792 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:58.775367 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:58.775383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.776366 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.776370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:58.777605 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:58.777617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:58.778641 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:58.778646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:58.778851 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:58.779634 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:46:58.779638 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:58.779691 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:46:58.779802 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:58.780241 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:58.780250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:58.780255 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:46:58.780396 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 15:46:58.780769 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260808 15:46:58.782199 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--builtin_ntp_servers=127.23.130.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:46:58.783787 11456 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37455
I20260808 15:46:58.784139 11456 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:58.785040 11456 heartbeater.cc:511] Master 127.23.130.62:37455 requested a full tablet report, sending...
I20260808 15:46:58.787403 11249 ts_manager.cc:194] Registered new tserver with Master: a5cd8b3bba69437988bc3fe25f89d7b0 (127.23.130.1:40723)
I20260808 15:46:58.789546 11249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.1:54735
W20260808 15:46:59.126920 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:59.127224 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:59.127370 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:59.134845 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:59.134949 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:59.135136 11460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:46:59.144408 11460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:59.145758 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:59.147643 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:59.161466 11470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:59.161624 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:59.161995 11468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:59.162504 11460 server_base.cc:1034] running on GCE node
I20260808 15:46:59.163152 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:59.164124 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:59.165378 11460 hybrid_clock.cc:648] HybridClock initialized: now 1786204019165324 us; error 45 us; skew 500 ppm
I20260808 15:46:59.165719 11460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:46:59.168102 11460 webserver.cc:460] Webserver started at http://127.23.130.2:45661/ using document root <none> and password file <none>
I20260808 15:46:59.168913 11460 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:59.169054 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:59.169406 11460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:59.172444 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "1ae3407866b04930b188923b11658197"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "9ccb67fc2632b1e97344ad0762136f33"
server_key_iv: "8293153619fc470f080582016cb60fc6"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.173287 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "1ae3407866b04930b188923b11658197"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "9ccb67fc2632b1e97344ad0762136f33"
server_key_iv: "8293153619fc470f080582016cb60fc6"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.178681 11460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260808 15:46:59.182174 11476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.183665 11460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:46:59.183832 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "1ae3407866b04930b188923b11658197"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "9ccb67fc2632b1e97344ad0762136f33"
server_key_iv: "8293153619fc470f080582016cb60fc6"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.184027 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:59.209326 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:59.210191 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:59.210568 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:59.211580 11460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:59.213158 11460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:59.213254 11460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.213356 11460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:59.213425 11460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.251753 11460 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:40275
I20260808 15:46:59.251816 11604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:40275 every 8 connection(s)
I20260808 15:46:59.253216 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:46:59.255566 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11460
I20260808 15:46:59.255755 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:46:59.256398 24072 external_mini_cluster.cc:1428] Setting key b6e14dd60c189bc3596e872d48394519
I20260808 15:46:59.259893 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--builtin_ntp_servers=127.23.130.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:46:59.269157 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.256210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:33569 (local address 127.23.130.62:37455)
0808 15:46:59.256489 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.256495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.257995 (+ 1500us) server_negotiation.cc:407] Connection header received
0808 15:46:59.260552 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.260557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.260629 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.260717 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:59.262247 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.263381 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.265943 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.266608 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.266716 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.268413 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:59.268418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:59.268421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:59.268440 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:59.268470 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:59.268482 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:59.268554 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.268832 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.268922 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 15:46:59.270779 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.256747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37455 (local address 127.23.130.2:33569)
0808 15:46:59.257841 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.257878 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.260232 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.261013 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.261029 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.261160 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:59.262043 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.262074 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.263554 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.263562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.265725 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.265743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.266852 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.266862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.267217 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.268023 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:46:59.268029 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:59.268086 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:46:59.268205 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:59.268585 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:59.268598 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:59.268608 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.268808 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.269284 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
I20260808 15:46:59.272411 11605 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37455
I20260808 15:46:59.272742 11605 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:59.273558 11605 heartbeater.cc:511] Master 127.23.130.62:37455 requested a full tablet report, sending...
I20260808 15:46:59.275142 11249 ts_manager.cc:194] Registered new tserver with Master: 1ae3407866b04930b188923b11658197 (127.23.130.2:40275)
I20260808 15:46:59.276383 11249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.2:33569
W20260808 15:46:59.608915 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:46:59.609203 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:46:59.609314 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:46:59.616778 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:46:59.616892 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:46:59.617000 11609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:46:59.626295 11609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:46:59.627728 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:46:59.629593 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:46:59.643157 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:59.643025 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:46:59.643015 11617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:46:59.644603 11609 server_base.cc:1034] running on GCE node
I20260808 15:46:59.645238 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:46:59.646225 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:46:59.647457 11609 hybrid_clock.cc:648] HybridClock initialized: now 1786204019647396 us; error 52 us; skew 500 ppm
I20260808 15:46:59.647774 11609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:46:59.650148 11609 webserver.cc:460] Webserver started at http://127.23.130.3:39053/ using document root <none> and password file <none>
I20260808 15:46:59.650908 11609 fs_manager.cc:359] Metadata directory not provided
I20260808 15:46:59.651029 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:46:59.651364 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:46:59.654359 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "9d518186aa59433fa58b10d61753f5b4"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "2f8ac911a609ad2b1665a9418a1774a9"
server_key_iv: "8fe3f12403a3406ea9abd9de78caad12"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.655233 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "9d518186aa59433fa58b10d61753f5b4"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "2f8ac911a609ad2b1665a9418a1774a9"
server_key_iv: "8fe3f12403a3406ea9abd9de78caad12"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.660194 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:46:59.663707 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.665150 11609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:46:59.665333 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "9d518186aa59433fa58b10d61753f5b4"
format_stamp: "Formatted at 2026-08-08 15:46:59 on dist-test-slave-gp6n"
server_key: "2f8ac911a609ad2b1665a9418a1774a9"
server_key_iv: "8fe3f12403a3406ea9abd9de78caad12"
server_key_version: "encryptionkey@0"
I20260808 15:46:59.665588 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:46:59.683588 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:46:59.684386 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:46:59.684702 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:46:59.685638 11609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:46:59.687278 11609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:46:59.687428 11609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.687551 11609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:46:59.687638 11609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:46:59.722110 11609 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:42903
I20260808 15:46:59.722199 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:42903 every 8 connection(s)
I20260808 15:46:59.723632 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:46:59.734006 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11609
I20260808 15:46:59.734200 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:46:59.734717 24072 external_mini_cluster.cc:1428] Setting key 05a0e33b8c2387013c4f836ba03d5e83
I20260808 15:46:59.736068 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.726120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:45189 (local address 127.23.130.62:37455)
0808 15:46:59.726345 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.726352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.727771 (+ 1419us) server_negotiation.cc:407] Connection header received
0808 15:46:59.729517 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.729521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.729591 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.729682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:59.730847 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.731620 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.733145 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.733529 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.733703 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.735176 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:59.735181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:59.735184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:59.735202 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:59.735235 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:59.735248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:59.735355 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.735605 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.735703 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:46:59.737004 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.726716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37455 (local address 127.23.130.3:45189)
0808 15:46:59.727616 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.727660 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.729258 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.729929 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.729938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.730024 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:46:59.730681 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.730701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.731759 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.731763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.733026 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.733038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.733758 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.733763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.733964 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.734832 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:46:59.734836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:46:59.734901 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:46:59.734980 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:46:59.735360 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:46:59.735373 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:46:59.735382 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.735571 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.735959 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260808 15:46:59.738289 11745 heartbeater.cc:347] Connected to a master server at 127.23.130.62:37455
I20260808 15:46:59.738608 11745 heartbeater.cc:464] Registering TS with master...
I20260808 15:46:59.739548 11745 heartbeater.cc:511] Master 127.23.130.62:37455 requested a full tablet report, sending...
I20260808 15:46:59.740921 11249 ts_manager.cc:194] Registered new tserver with Master: 9d518186aa59433fa58b10d61753f5b4 (127.23.130.3:42903)
I20260808 15:46:59.741698 11249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.3:45189
I20260808 15:46:59.749768 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:46:59.759856 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.753892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.23.130.62:37455)
0808 15:46:59.754090 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.754094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.754588 (+ 494us) server_negotiation.cc:407] Connection header received
0808 15:46:59.754724 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.754728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.754786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.754858 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:59.755307 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.756016 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.757381 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.757806 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.757951 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.759105 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:59.759110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:59.759113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:59.759129 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:59.759161 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:59.759173 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:59.759266 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.759447 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.759541 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:46:59.763679 11249 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:46:59.772954 11249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:46:59.793403 11456 heartbeater.cc:503] Master 127.23.130.62:37455 was elected leader, sending a full tablet report...
I20260808 15:46:59.796834 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.784692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:40275 (local address 127.0.0.1:39006)
0808 15:46:59.785608 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.785652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.785831 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.787609 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.787616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.787654 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.788117 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.788134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.790705 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.790709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.793840 (+ 3131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.793857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.795820 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.795826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.796039 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.796097 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.796235 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.796426 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":233,"thread_start_us":498,"threads_started":1}
I20260808 15:46:59.796892 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.784890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.23.130.2:40275)
0808 15:46:59.787090 (+ 2200us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.787107 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.787129 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:46:59.787211 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.787217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.787389 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.787947 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.788935 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.790560 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.794093 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.795625 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.795769 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.796037 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.796300 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.796503 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2053,"thread_start_us":1935,"threads_started":1}
I20260808 15:46:59.797639 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.785329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.23.130.1:40723)
0808 15:46:59.786945 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.786953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.786985 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:46:59.787083 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.787090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.787338 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.787577 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.789104 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.790474 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.794321 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.795789 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.795922 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.796155 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.797267 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.797400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1426,"thread_start_us":2149,"threads_started":1}
I20260808 15:46:59.798102 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.786743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:42903 (local address 127.0.0.1:46238)
0808 15:46:59.788727 (+ 1984us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.788745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.788864 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.790083 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.790089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.790115 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.790534 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.790551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.792314 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.792322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.794449 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.794463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.797343 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.797349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.797524 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.797555 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.797630 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.797739 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"mutex_wait_us":3,"thread_start_us":1702,"threads_started":1}
I20260808 15:46:59.798179 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.784880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48456)
0808 15:46:59.786173 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.786189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.786590 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.787840 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.787844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.787864 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.788747 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.788762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.790642 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.790649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.794067 (+ 3418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.794080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.796304 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.796311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.796517 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.796548 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.796640 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.796771 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":184,"thread_start_us":1044,"threads_started":1}
I20260808 15:46:59.799306 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.787309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.23.130.3:42903)
0808 15:46:59.789529 (+ 2220us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.789538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.789571 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:46:59.789653 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.789659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.789878 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.790068 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.790766 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.792178 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.795360 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.796817 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.798445 (+ 1628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.798713 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.798781 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.798945 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1995,"thread_start_us":1831,"threads_started":1}
I20260808 15:46:59.801041 11531 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5c94936604e07aa48be68593be5c5 (DEFAULT_TABLE table=test-table [id=414018c9845b45ec83c22ee4fc66a3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:59.802372 11531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5c94936604e07aa48be68593be5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:59.801964 11383 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5c94936604e07aa48be68593be5c5 (DEFAULT_TABLE table=test-table [id=414018c9845b45ec83c22ee4fc66a3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:59.802829 11676 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5c94936604e07aa48be68593be5c5 (DEFAULT_TABLE table=test-table [id=414018c9845b45ec83c22ee4fc66a3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:46:59.803380 11383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5c94936604e07aa48be68593be5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:59.804267 11676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5c94936604e07aa48be68593be5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:46:59.814203 11764 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Bootstrap starting.
I20260808 15:46:59.817476 11764 tablet_bootstrap.cc:654] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:59.818050 11766 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: Bootstrap starting.
I20260808 15:46:59.818961 11764 log.cc:824] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:59.820647 11767 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Bootstrap starting.
I20260808 15:46:59.822146 11766 tablet_bootstrap.cc:654] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:59.823323 11766 log.cc:824] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:59.823613 11767 tablet_bootstrap.cc:654] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Neither blocks nor log segments found. Creating new log.
I20260808 15:46:59.824851 11767 log.cc:824] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:46:59.825049 11764 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: No bootstrap required, opened a new log
I20260808 15:46:59.825318 11764 ts_tablet_manager.cc:1397] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260808 15:46:59.826383 11766 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: No bootstrap required, opened a new log
I20260808 15:46:59.826671 11766 ts_tablet_manager.cc:1397] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260808 15:46:59.827510 11767 tablet_bootstrap.cc:492] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: No bootstrap required, opened a new log
I20260808 15:46:59.827816 11767 ts_tablet_manager.cc:1397] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 15:46:59.834849 11767 raft_consensus.cc:348] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.835232 11767 raft_consensus.cc:374] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:59.835341 11767 raft_consensus.cc:729] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d518186aa59433fa58b10d61753f5b4, State: Initialized, Role: FOLLOWER
I20260808 15:46:59.835994 11767 consensus_queue.cc:260] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.835954 11764 raft_consensus.cc:348] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.836383 11764 raft_consensus.cc:374] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:59.836469 11764 raft_consensus.cc:729] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae3407866b04930b188923b11658197, State: Initialized, Role: FOLLOWER
I20260808 15:46:59.837039 11764 consensus_queue.cc:260] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.837266 11766 raft_consensus.cc:348] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.837872 11766 raft_consensus.cc:374] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:46:59.838006 11766 raft_consensus.cc:729] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cd8b3bba69437988bc3fe25f89d7b0, State: Initialized, Role: FOLLOWER
I20260808 15:46:59.838625 11745 heartbeater.cc:503] Master 127.23.130.62:37455 was elected leader, sending a full tablet report...
I20260808 15:46:59.838867 11766 consensus_queue.cc:260] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.839277 11767 ts_tablet_manager.cc:1428] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 15:46:59.840140 11605 heartbeater.cc:503] Master 127.23.130.62:37455 was elected leader, sending a full tablet report...
I20260808 15:46:59.841003 11764 ts_tablet_manager.cc:1428] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260808 15:46:59.842548 11766 ts_tablet_manager.cc:1428] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260808 15:46:59.845295 11772 raft_consensus.cc:482] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:59.845661 11772 raft_consensus.cc:504] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.847461 11772 leader_election.cc:288] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cd8b3bba69437988bc3fe25f89d7b0 (127.23.130.1:40723), 9d518186aa59433fa58b10d61753f5b4 (127.23.130.3:42903)
I20260808 15:46:59.854214 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.847817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:48087 (local address 127.23.130.1:40723)
0808 15:46:59.847984 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.847989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.848639 (+ 650us) server_negotiation.cc:407] Connection header received
0808 15:46:59.848832 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.848837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.848933 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.849031 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.849702 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.850514 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.852453 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.853322 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.853501 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.853570 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.853857 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.853981 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:46:59.854351 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.847784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.23.130.2:48087)
0808 15:46:59.848523 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.848545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.848683 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.849169 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.849172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.849193 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.849562 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.849571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.850646 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.850691 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.852269 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.852282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.853513 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.853519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.853696 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.853721 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.853824 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.853951 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":407,"threads_started":1}
I20260808 15:46:59.855947 11405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "1ae3407866b04930b188923b11658197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" is_pre_election: true
I20260808 15:46:59.856216 11771 raft_consensus.cc:482] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:46:59.856506 11405 raft_consensus.cc:2454] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae3407866b04930b188923b11658197 in term 0.
I20260808 15:46:59.856532 11771 raft_consensus.cc:504] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.857203 11479 leader_election.cc:302] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae3407866b04930b188923b11658197, a5cd8b3bba69437988bc3fe25f89d7b0; no voters:
I20260808 15:46:59.857708 11772 raft_consensus.cc:2781] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:46:59.857867 11772 raft_consensus.cc:482] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:46:59.857970 11772 raft_consensus.cc:3037] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:59.858520 11771 leader_election.cc:288] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae3407866b04930b188923b11658197 (127.23.130.2:40275), a5cd8b3bba69437988bc3fe25f89d7b0 (127.23.130.1:40723)
I20260808 15:46:59.861186 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.848941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:55851 (local address 127.23.130.3:42903)
0808 15:46:59.849195 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.849203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.854682 (+ 5479us) server_negotiation.cc:407] Connection header received
0808 15:46:59.854979 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.854984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.855067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.855129 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.855658 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.856813 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.858857 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.860258 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.860417 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.860464 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.860818 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.860959 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:46:59.861217 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.848645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:42903 (local address 127.23.130.2:55851)
0808 15:46:59.854577 (+ 5932us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.854615 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.854777 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.855319 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.855322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.855350 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.855502 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.855512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.857007 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.857014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.858693 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.858708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.860540 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.860544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.860664 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.860686 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.860790 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.860933 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5852,"thread_start_us":6211,"threads_started":1}
I20260808 15:46:59.861459 11772 raft_consensus.cc:504] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.862099 11772 leader_election.cc:288] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [CANDIDATE]: Term 1 election: Requested vote from peers a5cd8b3bba69437988bc3fe25f89d7b0 (127.23.130.1:40723), 9d518186aa59433fa58b10d61753f5b4 (127.23.130.3:42903)
I20260808 15:46:59.862308 11698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "1ae3407866b04930b188923b11658197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d518186aa59433fa58b10d61753f5b4" is_pre_election: true
I20260808 15:46:59.862867 11698 raft_consensus.cc:2454] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae3407866b04930b188923b11658197 in term 0.
I20260808 15:46:59.862942 11697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "1ae3407866b04930b188923b11658197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d518186aa59433fa58b10d61753f5b4"
I20260808 15:46:59.863221 11697 raft_consensus.cc:3037] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:59.866264 11697 raft_consensus.cc:2454] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae3407866b04930b188923b11658197 in term 1.
I20260808 15:46:59.866729 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.859838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:51277 (local address 127.23.130.2:40275)
0808 15:46:59.860012 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.860020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.860042 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:46:59.860215 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.860220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.860322 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.860390 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.861156 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.862215 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.865004 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.865743 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.865874 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.865918 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.866302 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.866495 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:46:59.866737 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.858689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:40275 (local address 127.23.130.3:51277)
0808 15:46:59.859916 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.859934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.860065 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.860558 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.860563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.860588 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.860947 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.860967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.862443 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.862449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.864816 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.864831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.865917 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.865923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.866132 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.866175 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.866291 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.866441 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099}
I20260808 15:46:59.866829 11405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "1ae3407866b04930b188923b11658197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0"
I20260808 15:46:59.867048 11405 raft_consensus.cc:3037] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:46:59.867595 11555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "9d518186aa59433fa58b10d61753f5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae3407866b04930b188923b11658197" is_pre_election: true
I20260808 15:46:59.868100 11555 raft_consensus.cc:2379] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d518186aa59433fa58b10d61753f5b4 in current term 1: Already voted for candidate 1ae3407866b04930b188923b11658197 in this term.
I20260808 15:46:59.868641 11479 leader_election.cc:302] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae3407866b04930b188923b11658197, 9d518186aa59433fa58b10d61753f5b4; no voters:
I20260808 15:46:59.869050 11772 raft_consensus.cc:2781] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:46:59.869534 11772 raft_consensus.cc:686] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae3407866b04930b188923b11658197, State: Running, Role: LEADER
I20260808 15:46:59.870151 11772 consensus_queue.cc:237] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } }
I20260808 15:46:59.870435 11405 raft_consensus.cc:2454] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae3407866b04930b188923b11658197 in term 1.
I20260808 15:46:59.872474 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.859211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.23.130.3:59873)
0808 15:46:59.862354 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0808 15:46:59.862375 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:46:59.862587 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:46:59.866899 (+ 4312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:46:59.866904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:46:59.866935 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:46:59.867118 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.867130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.868249 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.868257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.870216 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:46:59.870241 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.871768 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.871774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:46:59.871943 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.871975 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:46:59.872067 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:46:59.872198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3016,"thread_start_us":890,"threads_started":1}
I20260808 15:46:59.873111 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.859326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59873 (local address 127.23.130.1:40723)
0808 15:46:59.859487 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.859492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.862482 (+ 2990us) server_negotiation.cc:407] Connection header received
0808 15:46:59.866540 (+ 4058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.866545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.866619 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.866707 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:46:59.867304 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.868095 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.870372 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.871620 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.871784 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.871847 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.872687 (+ 840us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.872863 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:46:59.873621 11405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5c94936604e07aa48be68593be5c5" candidate_uuid: "9d518186aa59433fa58b10d61753f5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" is_pre_election: true
I20260808 15:46:59.873832 11405 raft_consensus.cc:2379] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d518186aa59433fa58b10d61753f5b4 in current term 1: Already voted for candidate 1ae3407866b04930b188923b11658197 in this term.
I20260808 15:46:59.874464 11626 leader_election.cc:302] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d518186aa59433fa58b10d61753f5b4; no voters: 1ae3407866b04930b188923b11658197, a5cd8b3bba69437988bc3fe25f89d7b0
I20260808 15:46:59.874837 11771 raft_consensus.cc:2726] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:46:59.875624 11249 catalog_manager.cc:5495] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae3407866b04930b188923b11658197 (127.23.130.2). New cstate: current_term: 1 leader_uuid: "1ae3407866b04930b188923b11658197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae3407866b04930b188923b11658197" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 40275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 } health_report { overall_health: UNKNOWN } } }
I20260808 15:46:59.903801 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.898326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.23.130.2:40275)
0808 15:46:59.898544 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.898549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.898567 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:46:59.898707 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.898711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.898805 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.898940 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:46:59.899264 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.900059 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.901470 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.901954 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.902051 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.902611 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:46:59.903171 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:46:59.903288 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.903385 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.903475 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:46:59.912261 11697 raft_consensus.cc:1261] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 1 FOLLOWER]: Refusing update from remote peer 1ae3407866b04930b188923b11658197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:59.912284 11405 raft_consensus.cc:1261] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 1 FOLLOWER]: Refusing update from remote peer 1ae3407866b04930b188923b11658197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:46:59.913132 11777 consensus_queue.cc:1035] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d518186aa59433fa58b10d61753f5b4" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:46:59.913460 11772 consensus_queue.cc:1035] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cd8b3bba69437988bc3fe25f89d7b0" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:46:59.929205 11784 mvcc.cc:205] Tried to move back new op lower bound from 7316291665550880768 to 7316291665394114560. Current Snapshot: MvccSnapshot[applied={T|T < 7316291665550880768}]
I20260808 15:46:59.936949 11246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:59.937222 11246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:46:59.941090 11246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210c07d9baf84228bbb78525ad651f52: Sending DeleteTablet for 3 replicas of tablet e7d5c94936604e07aa48be68593be5c5
I20260808 15:46:59.942199 11531 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5c94936604e07aa48be68593be5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:59 UTC) from {username='slave'} at 127.0.0.1:39006
I20260808 15:46:59.942379 11676 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5c94936604e07aa48be68593be5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:59 UTC) from {username='slave'} at 127.0.0.1:46238
I20260808 15:46:59.943106 11383 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5c94936604e07aa48be68593be5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:46:59 UTC) from {username='slave'} at 127.0.0.1:48456
I20260808 15:46:59.943693 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11317
I20260808 15:46:59.964250 11795 tablet_replica.cc:330] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0: stopping tablet replica
I20260808 15:46:59.964903 11795 raft_consensus.cc:2229] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:59.965586 11795 raft_consensus.cc:2258] T e7d5c94936604e07aa48be68593be5c5 P a5cd8b3bba69437988bc3fe25f89d7b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:59.968888 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:46:59.945318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.23.130.1:40723)
0808 15:46:59.945527 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:46:59.945537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:46:59.955621 (+ 10084us) server_negotiation.cc:407] Connection header received
0808 15:46:59.956681 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:46:59.956692 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:46:59.956825 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:46:59.956937 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:46:59.957806 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.959131 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.961610 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:46:59.962502 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:46:59.962632 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:46:59.966577 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:46:59.966592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:46:59.966616 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:46:59.967683 (+ 1067us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:46:59.967764 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:46:59.967795 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:46:59.967916 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:46:59.968240 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 15:46:59.968389 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:46:59.970355 11450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:46:59.971097 11794 tablet_replica.cc:330] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: stopping tablet replica
I20260808 15:46:59.971673 11794 raft_consensus.cc:2229] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:46:59.972313 11794 raft_consensus.cc:2258] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:59.973342 11793 tablet_replica.cc:330] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: stopping tablet replica
I20260808 15:46:59.973851 11793 raft_consensus.cc:2229] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:46:59.974922 11793 raft_consensus.cc:2258] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:46:59.977311 11793 ts_tablet_manager.cc:1905] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:59.978268 11794 ts_tablet_manager.cc:1905] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:46:59.983245 11794 ts_tablet_manager.cc:1918] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:59.983419 11794 log.cc:1196] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/e7d5c94936604e07aa48be68593be5c5
I20260808 15:46:59.983987 11794 ts_tablet_manager.cc:1939] T e7d5c94936604e07aa48be68593be5c5 P 9d518186aa59433fa58b10d61753f5b4: Deleting consensus metadata
I20260808 15:46:59.985050 11793 ts_tablet_manager.cc:1918] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:46:59.985229 11793 log.cc:1196] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/e7d5c94936604e07aa48be68593be5c5
I20260808 15:46:59.985824 11793 ts_tablet_manager.cc:1939] T e7d5c94936604e07aa48be68593be5c5 P 1ae3407866b04930b188923b11658197: Deleting consensus metadata
I20260808 15:46:59.986039 11232 catalog_manager.cc:4841] TS 9d518186aa59433fa58b10d61753f5b4 (127.23.130.3:42903): tablet e7d5c94936604e07aa48be68593be5c5 (table test-table [id=414018c9845b45ec83c22ee4fc66a3bd]) successfully deleted
I20260808 15:46:59.987771 11230 catalog_manager.cc:4841] TS 1ae3407866b04930b188923b11658197 (127.23.130.2:40275): tablet e7d5c94936604e07aa48be68593be5c5 (table test-table [id=414018c9845b45ec83c22ee4fc66a3bd]) successfully deleted
I20260808 15:47:00.072132 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11317
W20260808 15:47:00.093048 11232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:40723 (error 108)
I20260808 15:47:00.093891 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11460
I20260808 15:47:00.095351 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.094848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48476)
0808 15:47:00.095036 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.095177 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 15:47:00.095623 11232 catalog_manager.cc:4584] TS a5cd8b3bba69437988bc3fe25f89d7b0 (127.23.130.1:40723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7d5c94936604e07aa48be68593be5c5: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
I20260808 15:47:00.101509 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.095174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.23.130.2:40275)
0808 15:47:00.095330 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:00.095334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:00.095350 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:47:00.095424 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:00.095432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:00.095570 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:00.095700 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:00.096124 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:00.096890 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:00.098231 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:00.098834 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:00.098988 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:00.100105 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:00.100113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:00.100127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:00.100729 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:00.100781 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:00.100800 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:00.100923 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:00.101084 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 15:47:00.101184 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:47:00.102234 11600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:00.149427 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.148932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48482)
0808 15:47:00.149171 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.149258 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 15:47:00.150566 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.150154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48492)
0808 15:47:00.150303 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.150372 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:47:00.211455 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.210891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48502)
0808 15:47:00.211125 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.211224 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 15:47:00.212643 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.212275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48510)
0808 15:47:00.212444 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.212526 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:47:00.237957 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11460
I20260808 15:47:00.257419 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11609
I20260808 15:47:00.267107 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.259124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.23.130.3:42903)
0808 15:47:00.259361 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:00.259367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:00.259386 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:47:00.259450 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:00.259454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:00.259546 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:00.259663 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:00.260233 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:00.261287 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:00.263389 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:00.264146 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:00.264260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:00.265498 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:00.265511 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:00.265530 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:00.266155 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:00.266206 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:00.266232 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:00.266327 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:00.266616 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:47:00.266743 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 15:47:00.267915 11740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:00.293157 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.292619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48512)
0808 15:47:00.292850 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.292992 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 15:47:00.294342 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.293942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48516)
0808 15:47:00.294145 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.294223 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 15:47:00.359644 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11609
I20260808 15:47:00.379941 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11215
I20260808 15:47:00.381186 11285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:00.492770 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11215
I20260808 15:47:00.492827 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:00.491899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40723 (local address 127.0.0.1:48522)
0808 15:47:00.492129 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:00.492217 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
2026-08-08T15:47:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2884 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:00 dist-test-slave-gp6n krb5kdc[11804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:00 dist-test-slave-gp6n krb5kdc[11804](info): set up 2 sockets
Aug 08 15:47:00 dist-test-slave-gp6n krb5kdc[11804](info): commencing operation
krb5kdc: starting...
W20260808 15:47:01.676427 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:01 dist-test-slave-gp6n krb5kdc[11804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:01.982139 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34493
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:02.330118 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:02.330575 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:02.330753 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:02.338560 11819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:02.338656 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:02.338709 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:02.338786 11819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:02.338861 11819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:02.348601 11819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:02.349834 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:02.351189 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:02.363616 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:02.363677 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:02.363677 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:02.364588 11819 server_base.cc:1034] running on GCE node
I20260808 15:47:02.365319 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:02.366695 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:02.367928 11819 hybrid_clock.cc:648] HybridClock initialized: now 1786204022367894 us; error 41 us; skew 500 ppm
I20260808 15:47:02.368266 11819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:02 dist-test-slave-gp6n krb5kdc[11804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204022, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:02.371497 11819 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:02.372947 11819 webserver.cc:460] Webserver started at http://127.23.130.62:36895/ using document root <none> and password file <none>
I20260808 15:47:02.373631 11819 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:02.373780 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:02.374085 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:02.377027 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "d4f699c08001462db4dc8cf3a3839cbb"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "54017f37e5c78fcb1bc28ce998e2f2b0"
server_key_iv: "ed9be62925cf3c903e63e31b2fbf0fbf"
server_key_version: "encryptionkey@0"
I20260808 15:47:02.377767 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "d4f699c08001462db4dc8cf3a3839cbb"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "54017f37e5c78fcb1bc28ce998e2f2b0"
server_key_iv: "ed9be62925cf3c903e63e31b2fbf0fbf"
server_key_version: "encryptionkey@0"
I20260808 15:47:02.382822 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:47:02.386133 11836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:02.387409 11819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:02.387614 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "d4f699c08001462db4dc8cf3a3839cbb"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "54017f37e5c78fcb1bc28ce998e2f2b0"
server_key_iv: "ed9be62925cf3c903e63e31b2fbf0fbf"
server_key_version: "encryptionkey@0"
I20260808 15:47:02.387811 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:02.400686 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:02.404984 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:02.405269 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:02.424336 11819 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:34493
I20260808 15:47:02.424418 11894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:34493 every 8 connection(s)
I20260808 15:47:02.425697 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:02.429740 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:02.431600 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11819
I20260808 15:47:02.431783 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:02.432367 24072 external_mini_cluster.cc:1428] Setting key 7e2b551dcfeda5e131e8a6c3b2c8d89a
I20260808 15:47:02.442751 11895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: Bootstrap starting.
I20260808 15:47:02.445749 11895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: Neither blocks nor log segments found. Creating new log.
Aug 08 15:47:02 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:02.446897 11895 log.cc:824] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:02.450359 11895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: No bootstrap required, opened a new log
I20260808 15:47:02.455582 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:02.434342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.23.130.62:34493)
0808 15:47:02.435125 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:02.435136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:02.435175 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:47:02.436774 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:02.436798 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:02.437239 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:02.437620 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:02.439233 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:02.440572 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:02.441971 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:02.442695 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:02.442892 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:02.447328 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:02.447340 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:02.447364 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:02.447425 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:02.451373 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:02.452137 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:02.452145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:02.452156 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:02.452338 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:02.452787 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:02.452792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:02.452796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:02.453341 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:02.453676 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:02.454128 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 15:47:02.454611 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":345,"threads_started":1}
I20260808 15:47:02.458750 11895 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } }
I20260808 15:47:02.459125 11895 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:02.459240 11895 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f699c08001462db4dc8cf3a3839cbb, State: Initialized, Role: FOLLOWER
I20260808 15:47:02.459843 11895 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } }
I20260808 15:47:02.460103 11895 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:02.460255 11895 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:02.460464 11895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:02.462729 11895 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } }
I20260808 15:47:02.463207 11895 leader_election.cc:302] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f699c08001462db4dc8cf3a3839cbb; no voters:
I20260808 15:47:02.463789 11895 leader_election.cc:288] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:02.463928 11901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:02.464644 11901 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [term 1 LEADER]: Becoming Leader. State: Replica: d4f699c08001462db4dc8cf3a3839cbb, State: Running, Role: LEADER
I20260808 15:47:02.465219 11901 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } }
I20260808 15:47:02.465948 11895 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:02.469413 11903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f699c08001462db4dc8cf3a3839cbb. Latest consensus state: current_term: 1 leader_uuid: "d4f699c08001462db4dc8cf3a3839cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } } }
I20260808 15:47:02.469830 11903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:02.470446 11902 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f699c08001462db4dc8cf3a3839cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f699c08001462db4dc8cf3a3839cbb" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34493 } } }
I20260808 15:47:02.470942 11902 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:02.471379 11913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:02.476740 11913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:02.485294 11913 catalog_manager.cc:1347] Generated new cluster ID: f78550f3b8244d90ba30fe74c73935f2
I20260808 15:47:02.485412 11913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:02.497542 11913 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:02.499017 11913 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:02.505643 11913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: Generated new TSK 0
I20260808 15:47:02.506569 11913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:47:02.588275 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:02.945536 11925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:02.945835 11925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:02.946038 11925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:02.953410 11925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:02.953522 11925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:02.953699 11925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:47:02.963613 11925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:02.964973 11925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:02.966367 11925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:02.980504 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:02.980506 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:02.980681 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:02.981421 11925 server_base.cc:1034] running on GCE node
I20260808 15:47:02.982015 11925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:02.983014 11925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:02.984244 11925 hybrid_clock.cc:648] HybridClock initialized: now 1786204022984212 us; error 47 us; skew 500 ppm
I20260808 15:47:02.984598 11925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:02 dist-test-slave-gp6n krb5kdc[11804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204022, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:02.987954 11925 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:47:02.989454 11925 webserver.cc:460] Webserver started at http://127.23.130.1:40827/ using document root <none> and password file <none>
I20260808 15:47:02.990144 11925 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:02.990295 11925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:02.990686 11925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:02.994218 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "4a1b3a75b8444b9da53c08e106c9dee2"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "30da667c55138e56d758adb001a0eb88"
server_key_iv: "6a0520c47d895f65948d6016e018d0fe"
server_key_version: "encryptionkey@0"
I20260808 15:47:02.995146 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "4a1b3a75b8444b9da53c08e106c9dee2"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "30da667c55138e56d758adb001a0eb88"
server_key_iv: "6a0520c47d895f65948d6016e018d0fe"
server_key_version: "encryptionkey@0"
I20260808 15:47:03.000636 11925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260808 15:47:03.004612 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.006186 11925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:47:03.006387 11925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "4a1b3a75b8444b9da53c08e106c9dee2"
format_stamp: "Formatted at 2026-08-08 15:47:02 on dist-test-slave-gp6n"
server_key: "30da667c55138e56d758adb001a0eb88"
server_key_iv: "6a0520c47d895f65948d6016e018d0fe"
server_key_version: "encryptionkey@0"
I20260808 15:47:03.006678 11925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:03.039780 11925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:03.044792 11925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:03.045105 11925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:03.046109 11925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:03.047469 11925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:03.047576 11925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.047731 11925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:03.047806 11925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.081621 11925 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:43035
I20260808 15:47:03.081668 12065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:43035 every 8 connection(s)
I20260808 15:47:03.082962 11925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:47:03.085197 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 11925
I20260808 15:47:03.085474 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:47:03.086093 24072 external_mini_cluster.cc:1428] Setting key 1af04c567f39a47cfd72879a2b8ac1a2
Aug 08 15:47:03 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204022, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:03.113426 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:03.085708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:52317 (local address 127.23.130.62:34493)
0808 15:47:03.086295 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:03.086302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:03.087406 (+ 1104us) server_negotiation.cc:407] Connection header received
0808 15:47:03.089986 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:03.089991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:03.090053 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:03.090151 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:03.092378 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.093206 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.095741 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.096191 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.096311 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:03.107427 (+ 11116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:03.107433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:03.107436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:03.107471 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:03.110124 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:03.110989 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:03.110994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:03.110998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:03.111130 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:03.111602 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:03.111607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:03.111609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:03.111876 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:03.112040 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:03.112776 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 15:47:03.113014 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260808 15:47:03.114388 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:03.086332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34493 (local address 127.23.130.1:52317)
0808 15:47:03.087243 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:03.087298 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:03.089668 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:03.090459 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:03.090474 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:03.091358 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:03.092203 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:03.092232 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.093385 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.093393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:03.095572 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:03.095591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.096436 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.096447 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:03.096788 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:03.104480 (+ 7692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:03.104509 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:03.107139 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:03.110294 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:03.110304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:03.110322 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:03.110821 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:03.111252 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:03.111257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:03.111261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:03.111471 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:03.112013 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:03.112021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:03.112401 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 15:47:03.112680 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 15:47:03.113244 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":340,"threads_started":1}
I20260808 15:47:03.116123 12066 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34493
I20260808 15:47:03.116441 12066 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:03.117213 12066 heartbeater.cc:511] Master 127.23.130.62:34493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
I20260808 15:47:03.119642 11855 ts_manager.cc:194] Registered new tserver with Master: 4a1b3a75b8444b9da53c08e106c9dee2 (127.23.130.1:43035)
I20260808 15:47:03.121112 11855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:52317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:47:03.158278 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:03.502315 12072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:03.502703 12072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:03.502803 12072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:03.510273 12072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:03.510457 12072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:03.510596 12072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:47:03.520440 12072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:03.521651 12072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:03.523018 12072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:03.536516 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:03.536485 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:03.536569 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:03.537951 12072 server_base.cc:1034] running on GCE node
I20260808 15:47:03.538703 12072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:03.539675 12072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:03.540954 12072 hybrid_clock.cc:648] HybridClock initialized: now 1786204023540900 us; error 36 us; skew 500 ppm
I20260808 15:47:03.541317 12072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:03 dist-test-slave-gp6n krb5kdc[11804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204023, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:03.544746 12072 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:47:03.546219 12072 webserver.cc:460] Webserver started at http://127.23.130.2:46039/ using document root <none> and password file <none>
I20260808 15:47:03.546986 12072 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:03.547150 12072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:03.547503 12072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:03.550714 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "721ed2689b54469e9618b14cca0b7d4f"
format_stamp: "Formatted at 2026-08-08 15:47:03 on dist-test-slave-gp6n"
server_key: "efd0927e8eeeffa4ad6a797e1285e98c"
server_key_iv: "3b73517ba3940b3e85269e5ecc6c1819"
server_key_version: "encryptionkey@0"
I20260808 15:47:03.551569 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "721ed2689b54469e9618b14cca0b7d4f"
format_stamp: "Formatted at 2026-08-08 15:47:03 on dist-test-slave-gp6n"
server_key: "efd0927e8eeeffa4ad6a797e1285e98c"
server_key_iv: "3b73517ba3940b3e85269e5ecc6c1819"
server_key_version: "encryptionkey@0"
I20260808 15:47:03.557176 12072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:47:03.561225 12089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.562708 12072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:47:03.562902 12072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "721ed2689b54469e9618b14cca0b7d4f"
format_stamp: "Formatted at 2026-08-08 15:47:03 on dist-test-slave-gp6n"
server_key: "efd0927e8eeeffa4ad6a797e1285e98c"
server_key_iv: "3b73517ba3940b3e85269e5ecc6c1819"
server_key_version: "encryptionkey@0"
I20260808 15:47:03.563123 12072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:03.583141 12072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:03.587857 12072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:03.588188 12072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:03.589232 12072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:03.590798 12072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:03.590910 12072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.591109 12072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:03.591192 12072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:03.627578 12072 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:38393
I20260808 15:47:03.627624 12218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:38393 every 8 connection(s)
I20260808 15:47:03.628983 12072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:47:03.630048 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12072
I20260808 15:47:03.630263 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:47:03.630859 24072 external_mini_cluster.cc:1428] Setting key c5fab854a4c4d58e8740535438afc3a6
Aug 08 15:47:03 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204023, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:03.654919 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:03.632222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:47215 (local address 127.23.130.62:34493)
0808 15:47:03.632741 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:03.632748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:03.634020 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 15:47:03.636910 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:03.636915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:03.636983 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:03.637085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:03.639265 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.640411 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.642996 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.643730 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.643861 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:03.648412 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:03.648418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:03.648422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:03.648456 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:03.651426 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:03.652240 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:03.652246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:03.652255 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:03.652437 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:03.652930 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:03.652936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:03.652945 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:03.653239 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:03.653427 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:03.654208 (+ 781us) server_negotiation.cc:299] Negotiation successful
0808 15:47:03.654507 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260808 15:47:03.655987 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:03.632874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34493 (local address 127.23.130.2:47215)
0808 15:47:03.633791 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:03.633862 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:03.636587 (+ 2725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:03.637374 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:03.637387 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:03.638224 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:03.639068 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:03.639096 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.640586 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.640595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:03.642832 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:03.642850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:03.643988 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:03.643997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:03.644333 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:03.645152 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:03.645192 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:03.648142 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:03.651586 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:03.651596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:03.651615 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:03.652091 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:03.652569 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:03.652575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:03.652579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:03.652783 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:03.653382 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:03.653391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:03.653794 (+ 403us) client_negotiation.cc:769] Sending connection context
0808 15:47:03.654100 (+ 306us) client_negotiation.cc:240] Negotiation successful
0808 15:47:03.654709 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":383,"threads_started":1}
I20260808 15:47:03.657711 12219 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34493
I20260808 15:47:03.658046 12219 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:03.658887 12219 heartbeater.cc:511] Master 127.23.130.62:34493 requested a full tablet report, sending...
I20260808 15:47:03.660517 11855 ts_manager.cc:194] Registered new tserver with Master: 721ed2689b54469e9618b14cca0b7d4f (127.23.130.2:38393)
I20260808 15:47:03.661403 11855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:47215
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:47:03.702327 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:04.044576 12225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:04.044869 12225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:04.044965 12225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:04.052419 12225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:04.052524 12225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:04.052634 12225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:47:04.062510 12225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:34493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:04.063760 12225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:04.065004 12225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:04.079208 12231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:04.079308 12230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:04.079259 12233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:04.085160 12225 server_base.cc:1034] running on GCE node
I20260808 15:47:04.085865 12225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:04.086855 12225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:04.088091 12225 hybrid_clock.cc:648] HybridClock initialized: now 1786204024088058 us; error 66 us; skew 500 ppm
I20260808 15:47:04.088462 12225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:04 dist-test-slave-gp6n krb5kdc[11804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:04.091907 12225 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:47:04.093516 12225 webserver.cc:460] Webserver started at http://127.23.130.3:43865/ using document root <none> and password file <none>
I20260808 15:47:04.094261 12225 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:04.094411 12225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:04.094771 12225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:04.097719 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "916c461895e74cf6a18e4f9d77a3feb8"
format_stamp: "Formatted at 2026-08-08 15:47:04 on dist-test-slave-gp6n"
server_key: "b7e9f056a1f05d07d9b59dd722c80cf4"
server_key_iv: "f9de23b67355daf04d0315fb15469d97"
server_key_version: "encryptionkey@0"
I20260808 15:47:04.098594 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "916c461895e74cf6a18e4f9d77a3feb8"
format_stamp: "Formatted at 2026-08-08 15:47:04 on dist-test-slave-gp6n"
server_key: "b7e9f056a1f05d07d9b59dd722c80cf4"
server_key_iv: "f9de23b67355daf04d0315fb15469d97"
server_key_version: "encryptionkey@0"
I20260808 15:47:04.103919 12225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:47:04.107561 12240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:04.109055 12225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:47:04.109265 12225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "916c461895e74cf6a18e4f9d77a3feb8"
format_stamp: "Formatted at 2026-08-08 15:47:04 on dist-test-slave-gp6n"
server_key: "b7e9f056a1f05d07d9b59dd722c80cf4"
server_key_iv: "f9de23b67355daf04d0315fb15469d97"
server_key_version: "encryptionkey@0"
I20260808 15:47:04.109517 12225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:04.124071 12066 heartbeater.cc:503] Master 127.23.130.62:34493 was elected leader, sending a full tablet report...
I20260808 15:47:04.124322 12225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:04.129158 12225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:04.129539 12225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:04.130646 12225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:04.132094 12225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:04.132170 12225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:04.132256 12225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:04.132300 12225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:04.167603 12225 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:34885
I20260808 15:47:04.167649 12363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:34885 every 8 connection(s)
I20260808 15:47:04.169016 12225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:47:04.172679 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12225
I20260808 15:47:04.172885 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:47:04.173317 24072 external_mini_cluster.cc:1428] Setting key 9dc3da7c8bda772df39fb7fd08e226de
Aug 08 15:47:04 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:04.189884 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.171803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:42993 (local address 127.23.130.62:34493)
0808 15:47:04.172221 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.172226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.173470 (+ 1244us) server_negotiation.cc:407] Connection header received
0808 15:47:04.175891 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.175896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.175990 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.176106 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:04.177668 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.178531 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.180112 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.180507 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.180642 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.184024 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:04.184029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:04.184032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:04.184064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:04.186638 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.187452 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.187459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.187462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.187590 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.188040 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.188044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.188046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.188393 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:04.188585 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.189242 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.189469 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260808 15:47:04.190944 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.172394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34493 (local address 127.23.130.3:42993)
0808 15:47:04.173261 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.173308 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.175614 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.176358 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.176369 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.176937 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:04.177496 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.177514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.178687 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.178692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.179957 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.179969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.180740 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.180748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.180988 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.181599 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:04.181621 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:04.183804 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:04.186776 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:04.186785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:04.186799 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:04.187334 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:04.187704 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:04.187709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:04.187713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:04.187934 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:04.188546 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:04.188554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:04.188882 (+ 328us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.189155 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.189669 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":371,"threads_started":1}
I20260808 15:47:04.192333 12364 heartbeater.cc:347] Connected to a master server at 127.23.130.62:34493
I20260808 15:47:04.192662 12364 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:04.193346 12364 heartbeater.cc:511] Master 127.23.130.62:34493 requested a full tablet report, sending...
I20260808 15:47:04.194728 11855 ts_manager.cc:194] Registered new tserver with Master: 916c461895e74cf6a18e4f9d77a3feb8 (127.23.130.3:34885)
I20260808 15:47:04.195519 11855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:42993
I20260808 15:47:04.200403 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:47:04.219131 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.204637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.23.130.62:34493)
0808 15:47:04.204820 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.204828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.205300 (+ 472us) server_negotiation.cc:407] Connection header received
0808 15:47:04.205427 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.205430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.205511 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.205590 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:04.207067 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.207808 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.209128 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.209520 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.209612 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.212391 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:04.212396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:04.212399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:04.212429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:04.215419 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.216285 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.216290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.216296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.216493 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.217109 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.217116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.217122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.217625 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:04.217940 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.218465 (+ 525us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.218695 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:47:04.224128 11855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:47:04.231015 11855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:47:04.268950 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.243665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34885 (local address 127.0.0.1:33068)
0808 15:47:04.259024 (+ 15359us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.259085 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.259347 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.260741 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.260752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.260795 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:04.261414 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.261430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.263403 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.263408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.266062 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.266080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.267778 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.267785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.267976 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.268031 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.268217 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.268420 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14975,"mutex_wait_us":78,"thread_start_us":17724,"threads_started":1}
I20260808 15:47:04.269434 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.243815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.23.130.3:34885)
0808 15:47:04.257985 (+ 14170us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.257999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.259615 (+ 1616us) server_negotiation.cc:407] Connection header received
0808 15:47:04.259757 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.259768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.260121 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.260357 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:04.261685 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.263257 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.266242 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.267586 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.267723 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.268794 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.268876 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.269065 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13891,"thread_start_us":14208,"threads_started":1}
I20260808 15:47:04.269706 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.244239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.23.130.1:43035)
0808 15:47:04.260811 (+ 16572us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.260820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.260847 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:47:04.260943 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.260950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.261197 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.262074 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:04.262323 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.263897 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.266469 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.267980 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.268126 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.268385 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.269130 (+ 745us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.269369 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16392,"thread_start_us":16780,"threads_started":1}
I20260808 15:47:04.269610 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.243750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:43035 (local address 127.0.0.1:55818)
0808 15:47:04.260099 (+ 16349us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.260121 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.260243 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.261523 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.261528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.261561 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:04.262079 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.262092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.264062 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.264071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.266295 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.266305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.268549 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.268555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.268772 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.268809 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.268907 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.269130 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16229,"mutex_wait_us":1,"thread_start_us":14705,"threads_started":1}
I20260808 15:47:04.270514 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.247048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.23.130.2:38393)
0808 15:47:04.259668 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.259681 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.261871 (+ 2190us) server_negotiation.cc:407] Connection header received
0808 15:47:04.262065 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.262074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.262304 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.262520 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:04.263079 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.264568 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.266808 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.268239 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.268375 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.268565 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.270010 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.270182 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12463,"thread_start_us":12352,"threads_started":1}
I20260808 15:47:04.271216 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.246347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:38393 (local address 127.0.0.1:48350)
0808 15:47:04.261714 (+ 15367us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.261732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.261873 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.262536 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.262543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.262570 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:04.262898 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.262912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.264743 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.264754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.266631 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.266646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.269473 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.269482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.269668 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.269702 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.270701 (+ 999us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.270837 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15211,"mutex_wait_us":52,"thread_start_us":13483,"threads_started":1}
I20260808 15:47:04.273588 12293 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f (DEFAULT_TABLE table=test-table [id=cdb44690c3ee42ecb9343944073a2bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:04.273741 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f (DEFAULT_TABLE table=test-table [id=cdb44690c3ee42ecb9343944073a2bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:04.274910 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa5e1888ae46639a00a9e8c974de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:04.274474 12146 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f (DEFAULT_TABLE table=test-table [id=cdb44690c3ee42ecb9343944073a2bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:04.275206 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa5e1888ae46639a00a9e8c974de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:04.275784 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa5e1888ae46639a00a9e8c974de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:04.286653 12385 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Bootstrap starting.
I20260808 15:47:04.289820 12385 tablet_bootstrap.cc:654] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:04.290264 12386 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Bootstrap starting.
I20260808 15:47:04.291146 12387 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Bootstrap starting.
I20260808 15:47:04.291314 12385 log.cc:824] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:04.295120 12387 tablet_bootstrap.cc:654] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:04.295140 12386 tablet_bootstrap.cc:654] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:04.296249 12386 log.cc:824] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:04.296413 12387 log.cc:824] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:04.297303 12385 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: No bootstrap required, opened a new log
I20260808 15:47:04.297587 12385 ts_tablet_manager.cc:1397] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:47:04.299204 12387 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: No bootstrap required, opened a new log
I20260808 15:47:04.299461 12386 tablet_bootstrap.cc:492] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: No bootstrap required, opened a new log
I20260808 15:47:04.299471 12387 ts_tablet_manager.cc:1397] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260808 15:47:04.299712 12386 ts_tablet_manager.cc:1397] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:47:04.306967 12387 raft_consensus.cc:348] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.307369 12387 raft_consensus.cc:374] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:04.307504 12387 raft_consensus.cc:729] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721ed2689b54469e9618b14cca0b7d4f, State: Initialized, Role: FOLLOWER
I20260808 15:47:04.308060 12387 consensus_queue.cc:260] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.308470 12385 raft_consensus.cc:348] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.308950 12385 raft_consensus.cc:374] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:04.309079 12385 raft_consensus.cc:729] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916c461895e74cf6a18e4f9d77a3feb8, State: Initialized, Role: FOLLOWER
I20260808 15:47:04.309185 12219 heartbeater.cc:503] Master 127.23.130.62:34493 was elected leader, sending a full tablet report...
I20260808 15:47:04.309715 12385 consensus_queue.cc:260] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.309939 12387 ts_tablet_manager.cc:1428] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260808 15:47:04.310817 12386 raft_consensus.cc:348] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.311355 12386 raft_consensus.cc:374] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:04.311517 12386 raft_consensus.cc:729] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1b3a75b8444b9da53c08e106c9dee2, State: Initialized, Role: FOLLOWER
I20260808 15:47:04.312203 12385 ts_tablet_manager.cc:1428] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260808 15:47:04.312141 12386 consensus_queue.cc:260] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.313217 12364 heartbeater.cc:503] Master 127.23.130.62:34493 was elected leader, sending a full tablet report...
I20260808 15:47:04.314467 12386 ts_tablet_manager.cc:1428] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260808 15:47:04.335436 12067 tablet.cc:2369] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:47:04.381253 12220 tablet.cc:2369] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:47:04.420823 12365 tablet.cc:2369] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:04.660493 12392 raft_consensus.cc:482] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:47:04.660808 12392 raft_consensus.cc:504] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.662513 12392 leader_election.cc:288] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1b3a75b8444b9da53c08e106c9dee2 (127.23.130.1:43035), 721ed2689b54469e9618b14cca0b7d4f (127.23.130.2:38393)
I20260808 15:47:04.669122 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.662833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:43035 (local address 127.23.130.3:34357)
0808 15:47:04.663042 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.663065 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.663203 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.663942 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.663947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.663977 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:04.664138 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.664146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.665275 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.665280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.666930 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.666943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.668265 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.668272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.668475 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.668502 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.668621 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.668773 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:47:04.669122 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.663069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:34357 (local address 127.23.130.1:43035)
0808 15:47:04.663382 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.663393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.663424 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:47:04.663530 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.663540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.663632 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.663748 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:04.664305 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.665115 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.667124 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.668087 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.668269 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.668363 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.668650 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.668787 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:47:04.669508 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.663677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:60031 (local address 127.23.130.2:38393)
0808 15:47:04.663876 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.663883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.663903 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:47:04.664100 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.664106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.664174 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.664275 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:04.664800 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.665508 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.667586 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.668502 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.668709 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.668777 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.669112 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.669242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:47:04.670109 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.662902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:38393 (local address 127.23.130.3:60031)
0808 15:47:04.663813 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:04.663831 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:04.663950 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:04.664289 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:04.664293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:04.664316 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:04.664642 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.664651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.665670 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.665675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.667398 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:04.667410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.668722 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.668728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:04.668910 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.668972 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 15:47:04.669086 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:47:04.669797 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":3,"thread_start_us":636,"threads_started":1}
I20260808 15:47:04.670476 12017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa5e1888ae46639a00a9e8c974de2f" candidate_uuid: "916c461895e74cf6a18e4f9d77a3feb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" is_pre_election: true
I20260808 15:47:04.671026 12017 raft_consensus.cc:2454] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916c461895e74cf6a18e4f9d77a3feb8 in term 0.
I20260808 15:47:04.671100 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa5e1888ae46639a00a9e8c974de2f" candidate_uuid: "916c461895e74cf6a18e4f9d77a3feb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721ed2689b54469e9618b14cca0b7d4f" is_pre_election: true
I20260808 15:47:04.671649 12167 raft_consensus.cc:2454] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916c461895e74cf6a18e4f9d77a3feb8 in term 0.
I20260808 15:47:04.671878 12244 leader_election.cc:302] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1b3a75b8444b9da53c08e106c9dee2, 916c461895e74cf6a18e4f9d77a3feb8; no voters:
I20260808 15:47:04.672252 12392 raft_consensus.cc:2781] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:47:04.672456 12392 raft_consensus.cc:482] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:47:04.672577 12392 raft_consensus.cc:3037] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:04.675107 12392 raft_consensus.cc:504] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.675869 12392 leader_election.cc:288] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4a1b3a75b8444b9da53c08e106c9dee2 (127.23.130.1:43035), 721ed2689b54469e9618b14cca0b7d4f (127.23.130.2:38393)
I20260808 15:47:04.676272 12017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa5e1888ae46639a00a9e8c974de2f" candidate_uuid: "916c461895e74cf6a18e4f9d77a3feb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1b3a75b8444b9da53c08e106c9dee2"
I20260808 15:47:04.676529 12017 raft_consensus.cc:3037] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:04.676419 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa5e1888ae46639a00a9e8c974de2f" candidate_uuid: "916c461895e74cf6a18e4f9d77a3feb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721ed2689b54469e9618b14cca0b7d4f"
I20260808 15:47:04.676941 12167 raft_consensus.cc:3037] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:04.679101 12017 raft_consensus.cc:2454] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916c461895e74cf6a18e4f9d77a3feb8 in term 1.
I20260808 15:47:04.679276 12167 raft_consensus.cc:2454] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916c461895e74cf6a18e4f9d77a3feb8 in term 1.
I20260808 15:47:04.679536 12244 leader_election.cc:302] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1b3a75b8444b9da53c08e106c9dee2, 916c461895e74cf6a18e4f9d77a3feb8; no voters:
I20260808 15:47:04.679849 12392 raft_consensus.cc:2781] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:04.680344 12392 raft_consensus.cc:686] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 1 LEADER]: Becoming Leader. State: Replica: 916c461895e74cf6a18e4f9d77a3feb8, State: Running, Role: LEADER
I20260808 15:47:04.680855 12392 consensus_queue.cc:237] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } }
I20260808 15:47:04.686009 11854 catalog_manager.cc:5495] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 916c461895e74cf6a18e4f9d77a3feb8 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "916c461895e74cf6a18e4f9d77a3feb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916c461895e74cf6a18e4f9d77a3feb8" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 } health_report { overall_health: UNKNOWN } } }
I20260808 15:47:04.730304 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.724151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.23.130.3:34885)
0808 15:47:04.724413 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.724421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.724829 (+ 408us) server_negotiation.cc:407] Connection header received
0808 15:47:04.725093 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.725097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.725176 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.725337 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:04.725830 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.726625 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.728086 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.728490 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.728625 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.729190 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:04.729700 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:04.729911 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.730022 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.730070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:47:04.738658 12167 raft_consensus.cc:1261] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 1 FOLLOWER]: Refusing update from remote peer 916c461895e74cf6a18e4f9d77a3feb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:04.738659 12017 raft_consensus.cc:1261] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 1 FOLLOWER]: Refusing update from remote peer 916c461895e74cf6a18e4f9d77a3feb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:04.739643 12395 consensus_queue.cc:1035] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721ed2689b54469e9618b14cca0b7d4f" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:04.740033 12392 consensus_queue.cc:1035] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1b3a75b8444b9da53c08e106c9dee2" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:04.767091 11854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:04.767486 11854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:04.771541 11854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4f699c08001462db4dc8cf3a3839cbb: Sending DeleteTablet for 3 replicas of tablet 31aa5e1888ae46639a00a9e8c974de2f
I20260808 15:47:04.772717 12146 tablet_service.cc:1514] Processing DeleteTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:04 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:48350
I20260808 15:47:04.772717 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:04 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:55818
I20260808 15:47:04.772861 12293 tablet_service.cc:1514] Processing DeleteTablet for tablet 31aa5e1888ae46639a00a9e8c974de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:04 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:33068
I20260808 15:47:04.777915 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11925
I20260808 15:47:04.782934 12412 tablet_replica.cc:330] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: stopping tablet replica
I20260808 15:47:04.783530 12412 raft_consensus.cc:2229] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:47:04.784329 12412 raft_consensus.cc:2258] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:04.785190 12411 tablet_replica.cc:330] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: stopping tablet replica
I20260808 15:47:04.785593 12411 raft_consensus.cc:2229] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:04.786101 12411 raft_consensus.cc:2258] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:04.787093 12410 tablet_replica.cc:330] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: stopping tablet replica
I20260808 15:47:04.787482 12410 raft_consensus.cc:2229] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:04.787667 12412 ts_tablet_manager.cc:1905] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:04.787986 12410 raft_consensus.cc:2258] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:04.789520 12411 ts_tablet_manager.cc:1905] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:04.795220 12412 ts_tablet_manager.cc:1918] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:04.795423 12412 log.cc:1196] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/31aa5e1888ae46639a00a9e8c974de2f
I20260808 15:47:04.795900 12412 ts_tablet_manager.cc:1939] T 31aa5e1888ae46639a00a9e8c974de2f P 916c461895e74cf6a18e4f9d77a3feb8: Deleting consensus metadata
I20260808 15:47:04.796589 12411 ts_tablet_manager.cc:1918] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:04.796756 12411 log.cc:1196] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/31aa5e1888ae46639a00a9e8c974de2f
I20260808 15:47:04.797362 12411 ts_tablet_manager.cc:1939] T 31aa5e1888ae46639a00a9e8c974de2f P 721ed2689b54469e9618b14cca0b7d4f: Deleting consensus metadata
I20260808 15:47:04.799872 11840 catalog_manager.cc:4841] TS 916c461895e74cf6a18e4f9d77a3feb8 (127.23.130.3:34885): tablet 31aa5e1888ae46639a00a9e8c974de2f (table test-table [id=cdb44690c3ee42ecb9343944073a2bb3]) successfully deleted
I20260808 15:47:04.800594 12410 ts_tablet_manager.cc:1905] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:47:04 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:47:04.807050 11837 catalog_manager.cc:4841] TS 721ed2689b54469e9618b14cca0b7d4f (127.23.130.2:38393): tablet 31aa5e1888ae46639a00a9e8c974de2f (table test-table [id=cdb44690c3ee42ecb9343944073a2bb3]) successfully deleted
I20260808 15:47:04.807911 12410 ts_tablet_manager.cc:1918] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:04.808096 12410 log.cc:1196] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/31aa5e1888ae46639a00a9e8c974de2f
I20260808 15:47:04.808663 12410 ts_tablet_manager.cc:1939] T 31aa5e1888ae46639a00a9e8c974de2f P 4a1b3a75b8444b9da53c08e106c9dee2: Deleting consensus metadata
I20260808 15:47:04.810738 11839 catalog_manager.cc:4841] TS 4a1b3a75b8444b9da53c08e106c9dee2 (127.23.130.1:43035): tablet 31aa5e1888ae46639a00a9e8c974de2f (table test-table [id=cdb44690c3ee42ecb9343944073a2bb3]) successfully deleted
I20260808 15:47:04.811175 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.780004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.23.130.1:43035)
0808 15:47:04.780235 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.780244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.782052 (+ 1808us) server_negotiation.cc:407] Connection header received
0808 15:47:04.782146 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.782154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.782264 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.782422 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:04.792045 (+ 9623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.793208 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.798836 (+ 5628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.799390 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.799544 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.804002 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:04.804013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:04.804027 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:04.804064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:04.807674 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.808513 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.808520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.808523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.808667 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.809163 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.809168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.809171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.809539 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:04.809879 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.810496 (+ 617us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.810830 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:47:04.811872 12060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:04.902221 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11925
I20260808 15:47:04.924145 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12072
Aug 08 15:47:04 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:47:04.941833 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:04.925016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.23.130.2:38393)
0808 15:47:04.925222 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:04.925228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:04.925250 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:47:04.925386 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:04.925393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:04.925484 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:04.925603 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:04.927028 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.928159 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.929507 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:04.930169 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:04.930302 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:04.935064 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:04.935086 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:04.935108 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:04.935156 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:04.938942 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.939659 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.939669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.939674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.939840 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:04.940195 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:04.940200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:04.940204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:04.940709 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:04.941036 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:04.941206 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 15:47:04.941438 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:47:04.942625 12214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:05.032908 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12072
I20260808 15:47:05.054111 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12225
Aug 08 15:47:05 dist-test-slave-gp6n krb5kdc[11804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:47:05.074383 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:05.054993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.23.130.3:34885)
0808 15:47:05.055421 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:05.055425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:05.055440 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:47:05.055500 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:05.055507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:05.055566 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:05.055651 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:05.057608 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:05.058472 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:05.060398 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:05.060843 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:05.060931 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:05.067319 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:05.067333 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:05.067349 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:05.067386 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:05.071061 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:05.071940 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:05.071947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:05.071949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:05.072115 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:05.072599 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:05.072604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:05.072607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:05.073062 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:05.073350 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:05.073697 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 15:47:05.073890 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260808 15:47:05.075265 12358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:05.203384 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12225
I20260808 15:47:05.225651 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 11819
I20260808 15:47:05.226979 11890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:05.345501 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 11819
2026-08-08T15:47:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T15:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:05Z Disabled control of system clock
I20260808 15:47:05.438815 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38225
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:47:05.794950 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:05.795259 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:05.795353 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:05.802888 12421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:05.803010 12421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:05.803049 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:05.803107 12421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:05.803153 12421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:05.812636 12421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:38225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:05.813906 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:05.815634 12421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:05.827628 12427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:05.827680 12430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:05.827713 12428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:05.829162 12421 server_base.cc:1034] running on GCE node
I20260808 15:47:05.829988 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:05.831571 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:05.832814 12421 hybrid_clock.cc:648] HybridClock initialized: now 1786204025832772 us; error 39 us; skew 500 ppm
I20260808 15:47:05.833173 12421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:05.835557 12421 webserver.cc:460] Webserver started at http://127.23.130.62:40201/ using document root <none> and password file <none>
I20260808 15:47:05.836248 12421 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:05.836393 12421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:05.836725 12421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:05.839721 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "ca189d2195714935a0fde185bfd7c228"
format_stamp: "Formatted at 2026-08-08 15:47:05 on dist-test-slave-gp6n"
server_key: "ddfc83e75141cbd69e899c786fb75482"
server_key_iv: "0026f855d7baa08411aaada377a20733"
server_key_version: "encryptionkey@0"
I20260808 15:47:05.840547 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "ca189d2195714935a0fde185bfd7c228"
format_stamp: "Formatted at 2026-08-08 15:47:05 on dist-test-slave-gp6n"
server_key: "ddfc83e75141cbd69e899c786fb75482"
server_key_iv: "0026f855d7baa08411aaada377a20733"
server_key_version: "encryptionkey@0"
I20260808 15:47:05.845800 12421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:47:05.849254 12437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:05.850667 12421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:05.850857 12421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "ca189d2195714935a0fde185bfd7c228"
format_stamp: "Formatted at 2026-08-08 15:47:05 on dist-test-slave-gp6n"
server_key: "ddfc83e75141cbd69e899c786fb75482"
server_key_iv: "0026f855d7baa08411aaada377a20733"
server_key_version: "encryptionkey@0"
I20260808 15:47:05.851089 12421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:05.867269 12421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:05.868105 12421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:05.868409 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:05.887799 12421 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:38225
I20260808 15:47:05.887842 12491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:38225 every 8 connection(s)
I20260808 15:47:05.889107 12421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:05.889688 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12421
I20260808 15:47:05.889909 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:05.890585 24072 external_mini_cluster.cc:1428] Setting key f7d6a9cd7b6be1fcb4a3b652459d7ea8
I20260808 15:47:05.893838 12492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:05.907114 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:05.892624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.23.130.62:38225)
0808 15:47:05.893727 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:05.893742 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:05.893795 (+ 53us) server_negotiation.cc:407] Connection header received
0808 15:47:05.896157 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:05.896184 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:05.896783 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:05.897218 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:05.897763 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:05.899829 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:05.901481 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:05.902562 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:05.902737 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:05.904110 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:05.904122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:05.904140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:05.904899 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:05.904999 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:05.905031 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:05.905150 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:05.905428 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 15:47:05.905781 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":355,"threads_started":1}
I20260808 15:47:05.908121 12492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: Bootstrap starting.
I20260808 15:47:05.911759 12492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:05.912844 12492 log.cc:824] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:05.915824 12492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: No bootstrap required, opened a new log
I20260808 15:47:05.923024 12492 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } }
I20260808 15:47:05.923413 12492 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:05.923520 12492 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca189d2195714935a0fde185bfd7c228, State: Initialized, Role: FOLLOWER
I20260808 15:47:05.924117 12492 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } }
I20260808 15:47:05.924356 12492 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:05.924496 12492 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:05.924687 12492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:05.927047 12492 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } }
I20260808 15:47:05.927587 12492 leader_election.cc:302] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca189d2195714935a0fde185bfd7c228; no voters:
I20260808 15:47:05.928241 12492 leader_election.cc:288] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:05.928442 12497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:05.929179 12497 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [term 1 LEADER]: Becoming Leader. State: Replica: ca189d2195714935a0fde185bfd7c228, State: Running, Role: LEADER
I20260808 15:47:05.929736 12497 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } }
I20260808 15:47:05.930523 12492 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:05.932530 12498 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca189d2195714935a0fde185bfd7c228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } } }
I20260808 15:47:05.933009 12498 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:05.935509 12505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:05.935346 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca189d2195714935a0fde185bfd7c228. Latest consensus state: current_term: 1 leader_uuid: "ca189d2195714935a0fde185bfd7c228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca189d2195714935a0fde185bfd7c228" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 38225 } } }
I20260808 15:47:05.935657 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:05.940981 12505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:05.950672 12505 catalog_manager.cc:1347] Generated new cluster ID: f125a63c907f41939f619fe62d40c053
I20260808 15:47:05.950791 12505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:05.962111 12505 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:05.963423 12505 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:05.971051 12505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: Generated new TSK 0
I20260808 15:47:05.971848 12505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:05.982268 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--builtin_ntp_servers=127.23.130.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:47:06.340467 12519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:06.340758 12519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:06.340905 12519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:06.348820 12519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:06.348922 12519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:06.349100 12519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:47:06.358673 12519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:06.359997 12519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:06.361831 12519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:06.375442 12524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:06.375423 12527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:06.375375 12525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:06.376392 12519 server_base.cc:1034] running on GCE node
I20260808 15:47:06.377065 12519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:06.378051 12519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:06.379323 12519 hybrid_clock.cc:648] HybridClock initialized: now 1786204026379252 us; error 57 us; skew 500 ppm
I20260808 15:47:06.379683 12519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:06.382073 12519 webserver.cc:460] Webserver started at http://127.23.130.1:40021/ using document root <none> and password file <none>
I20260808 15:47:06.382862 12519 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:06.383002 12519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:06.383328 12519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:06.386286 12519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "1e061cdb43b940e085cf8331d3c092d6"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "8cb5ce64a6447306cf810c5dbf4f12fb"
server_key_iv: "25a9dae4c755b72f2b6225ec0d4da9f2"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.387172 12519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "1e061cdb43b940e085cf8331d3c092d6"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "8cb5ce64a6447306cf810c5dbf4f12fb"
server_key_iv: "25a9dae4c755b72f2b6225ec0d4da9f2"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.392479 12519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:06.396080 12533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.397491 12519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:47:06.397675 12519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "1e061cdb43b940e085cf8331d3c092d6"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "8cb5ce64a6447306cf810c5dbf4f12fb"
server_key_iv: "25a9dae4c755b72f2b6225ec0d4da9f2"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.397910 12519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:06.424815 12519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:06.425704 12519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:06.426034 12519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:06.427070 12519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:06.428606 12519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:06.428755 12519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.428898 12519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:06.428992 12519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.464259 12519 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:36069
I20260808 15:47:06.464377 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:36069 every 8 connection(s)
I20260808 15:47:06.465626 12519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:47:06.466995 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12519
I20260808 15:47:06.467192 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:47:06.467816 24072 external_mini_cluster.cc:1428] Setting key a69fe44e8c6e592ce5ab2677956538d1
I20260808 15:47:06.471418 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--builtin_ntp_servers=127.23.130.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:47:06.479212 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:06.468689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:43677 (local address 127.23.130.62:38225)
0808 15:47:06.469162 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:06.469168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:06.470622 (+ 1454us) server_negotiation.cc:407] Connection header received
0808 15:47:06.472604 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:06.472609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:06.472693 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:06.472818 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:06.473984 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.474752 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.476313 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.476727 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.476869 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:06.478252 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:06.478257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:06.478260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:06.478278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:06.478320 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:06.478332 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:06.478481 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:06.478732 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 15:47:06.478841 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260808 15:47:06.480185 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:06.469217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38225 (local address 127.23.130.1:43677)
0808 15:47:06.470358 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:06.470446 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:06.472320 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:06.473060 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:06.473068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:06.473154 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:06.473796 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:06.473819 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.474920 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.474923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:06.476160 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:06.476172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.476968 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.476976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:06.477201 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:06.477939 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:06.477942 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:06.477986 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:06.478058 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:06.478472 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:06.478484 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:06.478495 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:47:06.478710 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 15:47:06.479104 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":303,"threads_started":1}
I20260808 15:47:06.481691 12654 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38225
I20260808 15:47:06.482007 12654 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:06.482928 12654 heartbeater.cc:511] Master 127.23.130.62:38225 requested a full tablet report, sending...
I20260808 15:47:06.485095 12456 ts_manager.cc:194] Registered new tserver with Master: 1e061cdb43b940e085cf8331d3c092d6 (127.23.130.1:36069)
I20260808 15:47:06.486619 12456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.1:43677
W20260808 15:47:06.835909 12660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:06.836229 12660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:06.836390 12660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:06.843622 12660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:06.843755 12660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:06.843933 12660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:47:06.853039 12660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:06.854341 12660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:06.856668 12660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:06.870087 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:06.870150 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:06.870154 12671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:06.870951 12660 server_base.cc:1034] running on GCE node
I20260808 15:47:06.871562 12660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:06.872620 12660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:06.873857 12660 hybrid_clock.cc:648] HybridClock initialized: now 1786204026873823 us; error 56 us; skew 500 ppm
I20260808 15:47:06.874176 12660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:06.876602 12660 webserver.cc:460] Webserver started at http://127.23.130.2:40865/ using document root <none> and password file <none>
I20260808 15:47:06.877301 12660 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:06.877449 12660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:06.877771 12660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:06.880722 12660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "28ec7fc29a0d42a184884322ea4c3c20"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "bcc3d1aba96b9c6ec56cb7ec07334dc5"
server_key_iv: "1e9b75e2c94b2d7b53102932e9463e1a"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.881603 12660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "28ec7fc29a0d42a184884322ea4c3c20"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "bcc3d1aba96b9c6ec56cb7ec07334dc5"
server_key_iv: "1e9b75e2c94b2d7b53102932e9463e1a"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.886785 12660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:47:06.890368 12678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.891876 12660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:06.892084 12660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "28ec7fc29a0d42a184884322ea4c3c20"
format_stamp: "Formatted at 2026-08-08 15:47:06 on dist-test-slave-gp6n"
server_key: "bcc3d1aba96b9c6ec56cb7ec07334dc5"
server_key_iv: "1e9b75e2c94b2d7b53102932e9463e1a"
server_key_version: "encryptionkey@0"
I20260808 15:47:06.892395 12660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:06.907456 12660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:06.908285 12660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:06.908607 12660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:06.909646 12660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:06.911227 12660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:06.911371 12660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.911512 12660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:06.911607 12660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:06.947075 12660 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:38299
I20260808 15:47:06.947172 12809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:38299 every 8 connection(s)
I20260808 15:47:06.948454 12660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:47:06.953755 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12660
I20260808 15:47:06.953962 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:47:06.954610 24072 external_mini_cluster.cc:1428] Setting key 96e9fb818341b644ef469dc62d1967ef
I20260808 15:47:06.957928 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--builtin_ntp_servers=127.23.130.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:47:06.963901 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:06.951586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:36721 (local address 127.23.130.62:38225)
0808 15:47:06.951821 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:06.951827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:06.953261 (+ 1434us) server_negotiation.cc:407] Connection header received
0808 15:47:06.955770 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:06.955776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:06.955875 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:06.955977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:06.957356 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.958182 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.960754 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.961226 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.961342 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:06.963113 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:06.963118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:06.963122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:06.963140 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:06.963173 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:06.963186 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:06.963264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:06.963566 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 15:47:06.963650 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 15:47:06.965330 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:06.952100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38225 (local address 127.23.130.2:36721)
0808 15:47:06.953104 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:06.953142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:06.955506 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:06.956260 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:06.956271 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:06.956386 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:06.957155 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:06.957183 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.958336 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.958344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:06.960581 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:06.960600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:06.961465 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:06.961474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:06.961822 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:06.962731 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:06.962734 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:06.962803 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:06.962909 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:06.963290 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:06.963303 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:06.963313 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:47:06.963494 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 15:47:06.963928 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":294,"threads_started":1}
I20260808 15:47:06.967010 12810 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38225
I20260808 15:47:06.967374 12810 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:06.968438 12810 heartbeater.cc:511] Master 127.23.130.62:38225 requested a full tablet report, sending...
I20260808 15:47:06.970117 12456 ts_manager.cc:194] Registered new tserver with Master: 28ec7fc29a0d42a184884322ea4c3c20 (127.23.130.2:38299)
I20260808 15:47:06.971025 12456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.2:36721
W20260808 15:47:07.302990 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:07.303303 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:07.303498 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:07.310921 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:07.311017 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:07.311124 12813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:47:07.320237 12813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:07.321590 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:07.323496 12813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:07.336771 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:07.336913 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:07.336865 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:07.337673 12813 server_base.cc:1034] running on GCE node
I20260808 15:47:07.338241 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:07.339310 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:07.340562 12813 hybrid_clock.cc:648] HybridClock initialized: now 1786204027340527 us; error 91 us; skew 500 ppm
I20260808 15:47:07.340909 12813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:07.343307 12813 webserver.cc:460] Webserver started at http://127.23.130.3:44279/ using document root <none> and password file <none>
I20260808 15:47:07.344026 12813 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:07.344165 12813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:07.344498 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:07.347483 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "bdd6c37557d341cb824453712d552f01"
format_stamp: "Formatted at 2026-08-08 15:47:07 on dist-test-slave-gp6n"
server_key: "1d6c1631cbcfb3c62aaaf4403e3ca64e"
server_key_iv: "bd6587c741aa82518694c1bd98188bd6"
server_key_version: "encryptionkey@0"
I20260808 15:47:07.348284 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "bdd6c37557d341cb824453712d552f01"
format_stamp: "Formatted at 2026-08-08 15:47:07 on dist-test-slave-gp6n"
server_key: "1d6c1631cbcfb3c62aaaf4403e3ca64e"
server_key_iv: "bd6587c741aa82518694c1bd98188bd6"
server_key_version: "encryptionkey@0"
I20260808 15:47:07.353446 12813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:47:07.357241 12828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:07.358803 12813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:07.358991 12813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "bdd6c37557d341cb824453712d552f01"
format_stamp: "Formatted at 2026-08-08 15:47:07 on dist-test-slave-gp6n"
server_key: "1d6c1631cbcfb3c62aaaf4403e3ca64e"
server_key_iv: "bd6587c741aa82518694c1bd98188bd6"
server_key_version: "encryptionkey@0"
I20260808 15:47:07.359218 12813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:07.380414 12813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:07.381335 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:07.381690 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:07.382747 12813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:07.384379 12813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:07.384518 12813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:07.384658 12813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:07.384752 12813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:07.420073 12813 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:46073
I20260808 15:47:07.420168 12955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:46073 every 8 connection(s)
I20260808 15:47:07.421478 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:47:07.421934 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 12813
I20260808 15:47:07.422134 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:47:07.422803 24072 external_mini_cluster.cc:1428] Setting key 37463c1be1e599ec0080de6a14168c64
I20260808 15:47:07.434317 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.424251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:46863 (local address 127.23.130.62:38225)
0808 15:47:07.424468 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.424474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.426067 (+ 1593us) server_negotiation.cc:407] Connection header received
0808 15:47:07.427891 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.427895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.427959 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.428049 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:07.429072 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.429961 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.431679 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.432204 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.432351 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.433582 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:07.433588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:07.433591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:07.433609 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:07.433643 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:07.433655 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:07.433736 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.433968 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.434059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:47:07.435312 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.424845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:38225 (local address 127.23.130.3:46863)
0808 15:47:07.425909 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.425955 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.427647 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.428237 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.428245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.428328 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:07.428877 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.428917 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.430102 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.430105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.431545 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.431557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.432393 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.432399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.432610 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.433266 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:07.433268 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:07.433307 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:07.433386 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:07.433747 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:07.433755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:07.433760 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.433915 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.434201 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":347,"threads_started":1}
I20260808 15:47:07.436993 12956 heartbeater.cc:347] Connected to a master server at 127.23.130.62:38225
I20260808 15:47:07.437337 12956 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:07.438164 12956 heartbeater.cc:511] Master 127.23.130.62:38225 requested a full tablet report, sending...
I20260808 15:47:07.439589 12456 ts_manager.cc:194] Registered new tserver with Master: bdd6c37557d341cb824453712d552f01 (127.23.130.3:46073)
I20260808 15:47:07.440428 12456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.3:46863
I20260808 15:47:07.450599 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:47:07.461280 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.455072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.23.130.62:38225)
0808 15:47:07.455357 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.455366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.455684 (+ 318us) server_negotiation.cc:407] Connection header received
0808 15:47:07.455804 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.455808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.455878 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.455975 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:07.456427 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.457187 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.458681 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.459144 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.459247 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.460396 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:07.460401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:07.460405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:07.460476 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:07.460511 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:07.460524 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:07.460657 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.460887 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.460974 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:47:07.465754 12456 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:47:07.472872 12456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:47:07.489714 12654 heartbeater.cc:503] Master 127.23.130.62:38225 was elected leader, sending a full tablet report...
I20260808 15:47:07.497259 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.485196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:46073 (local address 127.0.0.1:54658)
0808 15:47:07.486044 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.486091 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.486260 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.488712 (+ 2452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.488722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.488758 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:07.489302 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.489319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.492552 (+ 3233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.492558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.494657 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.494670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.496397 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.496406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.496641 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.496739 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.496837 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.496989 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":1,"thread_start_us":434,"threads_started":1}
I20260808 15:47:07.498392 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.485433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.23.130.3:46073)
0808 15:47:07.487963 (+ 2530us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.487976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.488015 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:47:07.488117 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.488124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.488431 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.489796 (+ 1365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:07.490087 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.492373 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.494869 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.496160 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.497572 (+ 1412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.497857 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.497951 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.498101 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2303,"thread_start_us":2146,"threads_started":1}
I20260808 15:47:07.498857 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.486498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:38299 (local address 127.0.0.1:39860)
0808 15:47:07.488943 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.488967 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.489157 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.490460 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.490467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.490494 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:07.490932 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.490948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.492996 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.493005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.495447 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.495463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.497976 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.497984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.498177 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.498216 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.498311 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.498494 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"thread_start_us":2068,"threads_started":1}
I20260808 15:47:07.499178 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.486803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.23.130.2:38299)
0808 15:47:07.488371 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.488385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.489374 (+ 989us) server_negotiation.cc:407] Connection header received
0808 15:47:07.489522 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.489534 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.489862 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.490125 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:07.491152 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.492850 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.495709 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.497396 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.497545 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.497827 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.498474 (+ 647us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.498818 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1344,"thread_start_us":1186,"threads_started":1}
I20260808 15:47:07.499963 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.485892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58784)
0808 15:47:07.487159 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.487182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.487315 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.491222 (+ 3907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.491228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.491253 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:07.491641 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.491651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.493876 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.493882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.496396 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.496413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.499231 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.499240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.499439 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.499470 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.499544 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.499668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":944,"threads_started":1}
I20260808 15:47:07.500689 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.486633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.23.130.1:36069)
0808 15:47:07.490649 (+ 4016us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.490658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.490684 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:47:07.490771 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.490780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.491007 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.491857 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:07.492164 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.493722 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.496651 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.498397 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.499759 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.500059 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.500134 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.500321 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3832,"thread_start_us":3669,"threads_started":1}
I20260808 15:47:07.503052 12736 tablet_service.cc:1467] Processing CreateTablet for tablet 2896681e8cf9474bb8cd931d6031273f (DEFAULT_TABLE table=test-table [id=66f8021b2ae4433c82a8deffa11389e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:07.503077 12885 tablet_service.cc:1467] Processing CreateTablet for tablet 2896681e8cf9474bb8cd931d6031273f (DEFAULT_TABLE table=test-table [id=66f8021b2ae4433c82a8deffa11389e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:07.504459 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2896681e8cf9474bb8cd931d6031273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:07.504616 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2896681e8cf9474bb8cd931d6031273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:07.504635 12586 tablet_service.cc:1467] Processing CreateTablet for tablet 2896681e8cf9474bb8cd931d6031273f (DEFAULT_TABLE table=test-table [id=66f8021b2ae4433c82a8deffa11389e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:07.506070 12586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2896681e8cf9474bb8cd931d6031273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:07.515960 12976 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Bootstrap starting.
I20260808 15:47:07.519165 12976 tablet_bootstrap.cc:654] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:07.520500 12976 log.cc:824] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:07.521694 12977 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Bootstrap starting.
I20260808 15:47:07.524688 12979 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Bootstrap starting.
I20260808 15:47:07.526559 12977 tablet_bootstrap.cc:654] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:07.527657 12977 log.cc:824] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:07.528682 12979 tablet_bootstrap.cc:654] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:07.529440 12976 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: No bootstrap required, opened a new log
I20260808 15:47:07.529731 12976 ts_tablet_manager.cc:1397] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260808 15:47:07.529973 12979 log.cc:824] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:07.532284 12977 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: No bootstrap required, opened a new log
I20260808 15:47:07.532562 12977 ts_tablet_manager.cc:1397] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260808 15:47:07.533128 12979 tablet_bootstrap.cc:492] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: No bootstrap required, opened a new log
I20260808 15:47:07.533383 12979 ts_tablet_manager.cc:1397] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 15:47:07.540599 12976 raft_consensus.cc:348] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.541100 12976 raft_consensus.cc:374] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:07.541208 12976 raft_consensus.cc:729] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ec7fc29a0d42a184884322ea4c3c20, State: Initialized, Role: FOLLOWER
I20260808 15:47:07.540901 12977 raft_consensus.cc:348] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.541400 12977 raft_consensus.cc:374] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:07.541535 12977 raft_consensus.cc:729] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd6c37557d341cb824453712d552f01, State: Initialized, Role: FOLLOWER
I20260808 15:47:07.541833 12976 consensus_queue.cc:260] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.542186 12977 consensus_queue.cc:260] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.543195 12810 heartbeater.cc:503] Master 127.23.130.62:38225 was elected leader, sending a full tablet report...
I20260808 15:47:07.544292 12979 raft_consensus.cc:348] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.544723 12979 raft_consensus.cc:374] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:07.544814 12979 raft_consensus.cc:729] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e061cdb43b940e085cf8331d3c092d6, State: Initialized, Role: FOLLOWER
I20260808 15:47:07.545274 12976 ts_tablet_manager.cc:1428] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260808 15:47:07.545974 12979 consensus_queue.cc:260] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.547835 12979 ts_tablet_manager.cc:1428] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260808 15:47:07.548873 12956 heartbeater.cc:503] Master 127.23.130.62:38225 was elected leader, sending a full tablet report...
I20260808 15:47:07.549423 12977 ts_tablet_manager.cc:1428] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260808 15:47:07.636601 12984 raft_consensus.cc:482] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:47:07.636966 12984 raft_consensus.cc:504] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.638967 12984 leader_election.cc:288] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdd6c37557d341cb824453712d552f01 (127.23.130.3:46073), 28ec7fc29a0d42a184884322ea4c3c20 (127.23.130.2:38299)
I20260808 15:47:07.646831 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.639533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:54219 (local address 127.23.130.3:46073)
0808 15:47:07.639821 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.639828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.640153 (+ 325us) server_negotiation.cc:407] Connection header received
0808 15:47:07.640455 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.640463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.640562 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.640669 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:07.641495 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.642558 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.645107 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.645920 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.646117 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.646190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.646461 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.646590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:47:07.646996 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.639344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:46073 (local address 127.23.130.1:54219)
0808 15:47:07.640025 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.640042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.640229 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.640763 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.640768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.640820 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:07.641321 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.641337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.642692 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.642697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.644919 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.644931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.646118 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.646124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.646286 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.646312 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.646440 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.646570 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":113,"spinlock_wait_cycles":25984,"thread_start_us":410,"threads_started":1}
I20260808 15:47:07.647424 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.640247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:38299 (local address 127.23.130.1:39551)
0808 15:47:07.640899 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.640914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:07.641029 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:07.641484 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:07.641489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:07.641516 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:07.641896 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.641908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.643143 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.643149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.645187 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:07.645200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.646497 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.646505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:07.646681 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.646706 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:47:07.646853 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:47:07.647017 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":5,"thread_start_us":294,"threads_started":1}
I20260808 15:47:07.647676 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.640605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:39551 (local address 127.23.130.2:38299)
0808 15:47:07.640813 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.640819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.640999 (+ 180us) server_negotiation.cc:407] Connection header received
0808 15:47:07.641154 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.641159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.641251 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.641379 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:07.642061 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.642959 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.645400 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.646233 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.646471 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.646546 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.646886 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.647378 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:47:07.648120 12910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2896681e8cf9474bb8cd931d6031273f" candidate_uuid: "1e061cdb43b940e085cf8331d3c092d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd6c37557d341cb824453712d552f01" is_pre_election: true
I20260808 15:47:07.648650 12910 raft_consensus.cc:2454] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e061cdb43b940e085cf8331d3c092d6 in term 0.
I20260808 15:47:07.648650 12758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2896681e8cf9474bb8cd931d6031273f" candidate_uuid: "1e061cdb43b940e085cf8331d3c092d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec7fc29a0d42a184884322ea4c3c20" is_pre_election: true
I20260808 15:47:07.649250 12758 raft_consensus.cc:2454] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e061cdb43b940e085cf8331d3c092d6 in term 0.
I20260808 15:47:07.649384 12537 leader_election.cc:302] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e061cdb43b940e085cf8331d3c092d6, bdd6c37557d341cb824453712d552f01; no voters:
I20260808 15:47:07.649865 12984 raft_consensus.cc:2781] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:47:07.650090 12984 raft_consensus.cc:482] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:47:07.650225 12984 raft_consensus.cc:3037] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:07.652649 12984 raft_consensus.cc:504] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.653239 12984 leader_election.cc:288] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [CANDIDATE]: Term 1 election: Requested vote from peers bdd6c37557d341cb824453712d552f01 (127.23.130.3:46073), 28ec7fc29a0d42a184884322ea4c3c20 (127.23.130.2:38299)
I20260808 15:47:07.653854 12910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2896681e8cf9474bb8cd931d6031273f" candidate_uuid: "1e061cdb43b940e085cf8331d3c092d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd6c37557d341cb824453712d552f01"
I20260808 15:47:07.653895 12758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2896681e8cf9474bb8cd931d6031273f" candidate_uuid: "1e061cdb43b940e085cf8331d3c092d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec7fc29a0d42a184884322ea4c3c20"
I20260808 15:47:07.654075 12910 raft_consensus.cc:3037] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:07.654089 12758 raft_consensus.cc:3037] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:07.656525 12758 raft_consensus.cc:2454] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e061cdb43b940e085cf8331d3c092d6 in term 1.
I20260808 15:47:07.656801 12910 raft_consensus.cc:2454] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e061cdb43b940e085cf8331d3c092d6 in term 1.
I20260808 15:47:07.656982 12537 leader_election.cc:302] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e061cdb43b940e085cf8331d3c092d6, 28ec7fc29a0d42a184884322ea4c3c20; no voters:
I20260808 15:47:07.657307 12984 raft_consensus.cc:2781] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:07.657783 12984 raft_consensus.cc:686] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 1 LEADER]: Becoming Leader. State: Replica: 1e061cdb43b940e085cf8331d3c092d6, State: Running, Role: LEADER
I20260808 15:47:07.658296 12984 consensus_queue.cc:237] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } }
I20260808 15:47:07.663539 12455 catalog_manager.cc:5495] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e061cdb43b940e085cf8331d3c092d6 (127.23.130.1). New cstate: current_term: 1 leader_uuid: "1e061cdb43b940e085cf8331d3c092d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e061cdb43b940e085cf8331d3c092d6" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 36069 } health_report { overall_health: HEALTHY } } }
W20260808 15:47:07.673370 12958 tablet.cc:2369] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:07.694490 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.689117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.23.130.1:36069)
0808 15:47:07.689352 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.689358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.689379 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:47:07.689526 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.689530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.689634 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.689778 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:07.690124 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.691031 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.692423 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.692873 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.692981 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.693419 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:07.693930 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:07.694064 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.694171 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.694209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 15:47:07.700769 12811 tablet.cc:2369] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:07.703258 12910 raft_consensus.cc:1261] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 1 FOLLOWER]: Refusing update from remote peer 1e061cdb43b940e085cf8331d3c092d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:07.703332 12758 raft_consensus.cc:1261] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 1 FOLLOWER]: Refusing update from remote peer 1e061cdb43b940e085cf8331d3c092d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:07.704331 12984 consensus_queue.cc:1035] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd6c37557d341cb824453712d552f01" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:07.704725 12987 consensus_queue.cc:1035] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ec7fc29a0d42a184884322ea4c3c20" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:07.732882 12455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:07.733174 12455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:07.737171 12455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca189d2195714935a0fde185bfd7c228: Sending DeleteTablet for 3 replicas of tablet 2896681e8cf9474bb8cd931d6031273f
I20260808 15:47:07.738274 12885 tablet_service.cc:1514] Processing DeleteTablet for tablet 2896681e8cf9474bb8cd931d6031273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:07 UTC) from {username='slave'} at 127.0.0.1:54658
I20260808 15:47:07.738638 12736 tablet_service.cc:1514] Processing DeleteTablet for tablet 2896681e8cf9474bb8cd931d6031273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:07 UTC) from {username='slave'} at 127.0.0.1:39860
I20260808 15:47:07.739108 12586 tablet_service.cc:1514] Processing DeleteTablet for tablet 2896681e8cf9474bb8cd931d6031273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:07 UTC) from {username='slave'} at 127.0.0.1:58784
I20260808 15:47:07.741232 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12519
I20260808 15:47:07.755594 13008 tablet_replica.cc:330] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: stopping tablet replica
I20260808 15:47:07.756322 13008 raft_consensus.cc:2229] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:47:07.757170 13008 raft_consensus.cc:2258] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:07.759670 13008 ts_tablet_manager.cc:1905] T 2896681e8cf9474bb8cd931d6031273f P 1e061cdb43b940e085cf8331d3c092d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:07.761214 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.742702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.23.130.1:36069)
0808 15:47:07.742952 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.742961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.747019 (+ 4058us) server_negotiation.cc:407] Connection header received
0808 15:47:07.747931 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.747940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.748076 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.748169 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:07.749558 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.751200 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.756206 (+ 5006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.756889 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.756994 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.759304 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:07.759316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:07.759332 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:07.760056 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:07.760120 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:07.760144 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:07.760223 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.760709 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.760807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:47:07.762387 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:07.763085 13007 tablet_replica.cc:330] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: stopping tablet replica
I20260808 15:47:07.763940 13006 tablet_replica.cc:330] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: stopping tablet replica
I20260808 15:47:07.764282 13007 raft_consensus.cc:2229] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:07.764560 13006 raft_consensus.cc:2229] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:07.764856 13007 raft_consensus.cc:2258] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:07.765198 13006 raft_consensus.cc:2258] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:07.767140 13007 ts_tablet_manager.cc:1905] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:07.767933 13006 ts_tablet_manager.cc:1905] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:07.772753 13006 ts_tablet_manager.cc:1918] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:07.772931 13006 log.cc:1196] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/2896681e8cf9474bb8cd931d6031273f
I20260808 15:47:07.773432 13006 ts_tablet_manager.cc:1939] T 2896681e8cf9474bb8cd931d6031273f P bdd6c37557d341cb824453712d552f01: Deleting consensus metadata
I20260808 15:47:07.774423 13007 ts_tablet_manager.cc:1918] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:07.774600 13007 log.cc:1196] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/2896681e8cf9474bb8cd931d6031273f
I20260808 15:47:07.775180 13007 ts_tablet_manager.cc:1939] T 2896681e8cf9474bb8cd931d6031273f P 28ec7fc29a0d42a184884322ea4c3c20: Deleting consensus metadata
I20260808 15:47:07.775223 12440 catalog_manager.cc:4841] TS bdd6c37557d341cb824453712d552f01 (127.23.130.3:46073): tablet 2896681e8cf9474bb8cd931d6031273f (table test-table [id=66f8021b2ae4433c82a8deffa11389e5]) successfully deleted
I20260808 15:47:07.776952 12440 catalog_manager.cc:4841] TS 28ec7fc29a0d42a184884322ea4c3c20 (127.23.130.2:38299): tablet 2896681e8cf9474bb8cd931d6031273f (table test-table [id=66f8021b2ae4433c82a8deffa11389e5]) successfully deleted
I20260808 15:47:07.863427 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12519
I20260808 15:47:07.885298 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12660
W20260808 15:47:07.885322 12441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.130.1:36069 (error 108)
I20260808 15:47:07.887336 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.886834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58808)
0808 15:47:07.887009 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.887151 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 15:47:07.888319 12441 catalog_manager.cc:4584] TS 1e061cdb43b940e085cf8331d3c092d6 (127.23.130.1:36069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2896681e8cf9474bb8cd931d6031273f: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
I20260808 15:47:07.892952 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.886147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.23.130.2:38299)
0808 15:47:07.886313 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:07.886321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:07.886416 (+ 95us) server_negotiation.cc:407] Connection header received
0808 15:47:07.886620 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:07.886626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:07.886716 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:07.886827 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:07.887274 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.888102 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.889918 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:07.890318 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:07.890498 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:07.891549 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:07.891558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:07.891572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:07.892173 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:07.892239 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:07.892259 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:07.892342 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:07.892484 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 15:47:07.892569 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:47:07.893663 12804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:07.942010 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.941506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58814)
0808 15:47:07.941725 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.941820 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 15:47:07.943179 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:07.942799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58824)
0808 15:47:07.942957 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:07.943040 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:47:07.987282 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12660
I20260808 15:47:08.003891 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:08.003389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58832)
0808 15:47:08.003619 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:08.003716 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:47:08.005194 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:08.004703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58842)
0808 15:47:08.004987 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:08.005065 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 15:47:08.008594 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12813
I20260808 15:47:08.015770 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:08.009447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.23.130.3:46073)
0808 15:47:08.009626 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:08.009631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:08.009651 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:47:08.009857 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:08.009862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:08.009954 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:08.010042 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:08.010500 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:08.011277 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:08.012595 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:08.013011 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:08.013125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:08.014295 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:08.014304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:08.014318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:08.015007 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:08.015057 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:08.015078 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:08.015173 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:08.015332 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 15:47:08.015427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:47:08.016489 12951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:08.086017 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:08.085460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58848)
0808 15:47:08.085712 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:08.085827 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 15:47:08.087221 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:08.086791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:36069 (local address 127.0.0.1:58852)
0808 15:47:08.087016 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:08.087088 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 15:47:08.106936 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12813
I20260808 15:47:08.126248 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 12421
I20260808 15:47:08.127458 12487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:08.242091 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 12421
2026-08-08T15:47:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2896 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:08 dist-test-slave-gp6n krb5kdc[13021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:08 dist-test-slave-gp6n krb5kdc[13021](info): set up 2 sockets
Aug 08 15:47:08 dist-test-slave-gp6n krb5kdc[13021](info): commencing operation
krb5kdc: starting...
W20260808 15:47:09.505705 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:09 dist-test-slave-gp6n krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:09.819409 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42857
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:42857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:10.174383 13034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:10.174817 13034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:10.174986 13034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:10.182356 13034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:10.182569 13034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:10.182687 13034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:10.182806 13034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:10.182898 13034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:10.192950 13034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:42857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:10.194319 13034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:10.195727 13034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:10.207960 13039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:10.208067 13042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:10.208011 13040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:10.209312 13034 server_base.cc:1034] running on GCE node
I20260808 15:47:10.210346 13034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:10.211786 13034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:10.213011 13034 hybrid_clock.cc:648] HybridClock initialized: now 1786204030212957 us; error 35 us; skew 500 ppm
I20260808 15:47:10.213313 13034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:10 dist-test-slave-gp6n krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:10.216676 13034 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:10.218773 13034 webserver.cc:460] Webserver started at http://127.23.130.62:42607/ using document root <none> and password file <none>
I20260808 15:47:10.219735 13034 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:10.219900 13034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:10.220288 13034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:10.224586 13034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "b732611792ee6fa6bee669cabe6bc222"
server_key_iv: "4a42f2cd26dbcc9d68b1e41b22237be2"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.225692 13034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "b732611792ee6fa6bee669cabe6bc222"
server_key_iv: "4a42f2cd26dbcc9d68b1e41b22237be2"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.231463 13034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:47:10.234972 13049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:10.236336 13034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:47:10.236475 13034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "b732611792ee6fa6bee669cabe6bc222"
server_key_iv: "4a42f2cd26dbcc9d68b1e41b22237be2"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.236637 13034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:10.273448 13034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:10.278560 13034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:10.278904 13034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:10.301525 13034 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:42857
I20260808 15:47:10.301612 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:42857 every 8 connection(s)
I20260808 15:47:10.303102 13034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:10.307217 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:10.310866 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13034
I20260808 15:47:10.311092 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:10.311563 24072 external_mini_cluster.cc:1428] Setting key 9d184b3db8c4458c94cc43e09441e808
I20260808 15:47:10.320878 13110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: Bootstrap starting.
I20260808 15:47:10.324489 13110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:10.325536 13110 log.cc:824] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:47:10 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:10.328861 13110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: No bootstrap required, opened a new log
I20260808 15:47:10.336261 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:10.313540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.23.130.62:42857)
0808 15:47:10.314394 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:10.314440 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:10.314494 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:47:10.316531 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:10.316566 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:10.317098 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:10.317565 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:10.319011 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.321046 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.322327 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.323434 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.323624 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:10.328100 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:10.328110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:10.328129 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:10.328180 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:10.331805 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:10.332779 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:10.332786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:10.332791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:10.332980 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:10.333450 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:10.333454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:10.333456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:10.334026 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:10.334262 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:10.334745 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 15:47:10.335175 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":347,"threads_started":1}
I20260808 15:47:10.338260 13110 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } }
I20260808 15:47:10.338724 13110 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:10.338858 13110 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc41dc2cbb14b3fb07320d1bb437bd8, State: Initialized, Role: FOLLOWER
I20260808 15:47:10.339562 13110 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } }
I20260808 15:47:10.339867 13110 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:10.340034 13110 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:10.340205 13110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:10.343024 13110 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } }
I20260808 15:47:10.343549 13110 leader_election.cc:302] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc41dc2cbb14b3fb07320d1bb437bd8; no voters:
I20260808 15:47:10.344197 13110 leader_election.cc:288] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:10.344300 13115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:10.346369 13110 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:10.346907 13115 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [term 1 LEADER]: Becoming Leader. State: Replica: ddc41dc2cbb14b3fb07320d1bb437bd8, State: Running, Role: LEADER
I20260808 15:47:10.347415 13115 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } }
I20260808 15:47:10.350622 13117 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc41dc2cbb14b3fb07320d1bb437bd8. Latest consensus state: current_term: 1 leader_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } } }
I20260808 15:47:10.351038 13117 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:10.350329 13116 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc41dc2cbb14b3fb07320d1bb437bd8" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 42857 } } }
I20260808 15:47:10.351698 13116 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:10.352423 13127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:10.362886 13127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:10.371500 13127 catalog_manager.cc:1347] Generated new cluster ID: 29f550b3cd4649ae86d48b6fa9c62761
I20260808 15:47:10.371618 13127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:10.390130 13127 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:10.391544 13127 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:10.401453 13127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: Generated new TSK 0
I20260808 15:47:10.402556 13127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
Entry for principal kudu/127.23.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab.
I20260808 15:47:10.478267 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:10.816725 13137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:10.817022 13137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:10.817186 13137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:10.824573 13137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:10.824692 13137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:10.824870 13137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:47:10.834851 13137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.1.keytab
--principal=kudu/127.23.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:10.836231 13137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:10.837716 13137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:10.851603 13142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:10.851573 13146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:10.851655 13143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:10.852871 13137 server_base.cc:1034] running on GCE node
I20260808 15:47:10.853494 13137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:10.854547 13137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:10.855782 13137 hybrid_clock.cc:648] HybridClock initialized: now 1786204030855724 us; error 60 us; skew 500 ppm
I20260808 15:47:10.856117 13137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:10 dist-test-slave-gp6n krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:10.859377 13137 init.cc:377] Logged in from keytab as kudu/127.23.130.1@KRBTEST.COM (short username kudu)
I20260808 15:47:10.860898 13137 webserver.cc:460] Webserver started at http://127.23.130.1:35093/ using document root <none> and password file <none>
I20260808 15:47:10.861594 13137 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:10.861740 13137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:10.862068 13137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:10.864995 13137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "86327eef1c5e497db26648710e7f5ed9"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "c1dbc22148f93aa287c7c9fcc7c53e72"
server_key_iv: "382b1a0d5f2d9516ea118536439d0285"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.865773 13137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "86327eef1c5e497db26648710e7f5ed9"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "c1dbc22148f93aa287c7c9fcc7c53e72"
server_key_iv: "382b1a0d5f2d9516ea118536439d0285"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.871019 13137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:47:10.874396 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:10.875765 13137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:47:10.875953 13137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "86327eef1c5e497db26648710e7f5ed9"
format_stamp: "Formatted at 2026-08-08 15:47:10 on dist-test-slave-gp6n"
server_key: "c1dbc22148f93aa287c7c9fcc7c53e72"
server_key_iv: "382b1a0d5f2d9516ea118536439d0285"
server_key_version: "encryptionkey@0"
I20260808 15:47:10.876154 13137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:10.889509 13137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:10.893447 13137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:10.893738 13137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:10.894739 13137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:10.896073 13137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:10.896183 13137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:10.896332 13137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:10.896411 13137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:10.932300 13137 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:40923
I20260808 15:47:10.932333 13276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:40923 every 8 connection(s)
I20260808 15:47:10.933605 13137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:47:10.938558 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13137
I20260808 15:47:10.938794 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:47:10.939323 24072 external_mini_cluster.cc:1428] Setting key ebf1e80b62d31088adede3d6edef1458
Aug 08 15:47:10 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.1@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:10.954738 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:10.936391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:44735 (local address 127.23.130.62:42857)
0808 15:47:10.936839 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:10.936844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:10.937907 (+ 1063us) server_negotiation.cc:407] Connection header received
0808 15:47:10.940126 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:10.940132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:10.940226 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:10.940347 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:10.941992 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.942778 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.944329 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.944736 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.944871 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:10.948637 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:10.948642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:10.948646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:10.948679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:10.951584 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:10.952314 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:10.952320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:10.952323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:10.952463 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:10.952927 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:10.952932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:10.952935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:10.953216 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:10.953385 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:10.954104 (+ 719us) server_negotiation.cc:299] Negotiation successful
0808 15:47:10.954333 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260808 15:47:10.955786 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:10.937022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42857 (local address 127.23.130.1:44735)
0808 15:47:10.937747 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:10.937785 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:10.939811 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:10.940700 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:10.940709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:10.941237 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:10.941815 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:10.941833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.942926 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.942931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:10.944179 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:10.944191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:10.944975 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:10.944984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:10.945240 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:10.946004 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:10.946026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:10.948385 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:10.951737 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:10.951746 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:10.951763 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:10.952187 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:10.952588 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:10.952593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:10.952597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:10.952802 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:10.953355 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:10.953363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:10.953725 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 15:47:10.954002 (+ 277us) client_negotiation.cc:240] Negotiation successful
0808 15:47:10.954583 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":285,"threads_started":1}
I20260808 15:47:10.957401 13277 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42857
I20260808 15:47:10.957770 13277 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:10.958812 13277 heartbeater.cc:511] Master 127.23.130.62:42857 requested a full tablet report, sending...
I20260808 15:47:10.961227 13070 ts_manager.cc:194] Registered new tserver with Master: 86327eef1c5e497db26648710e7f5ed9 (127.23.130.1:40923)
I20260808 15:47:10.962782 13070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.1@KRBTEST.COM'} at 127.23.130.1:44735
WARNING: no policy specified for kudu/127.23.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
Entry for principal kudu/127.23.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab.
I20260808 15:47:11.019423 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:11.360649 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:11.360998 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:11.361171 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:11.369036 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:11.369233 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:11.369455 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:47:11.379451 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.2.keytab
--principal=kudu/127.23.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:11.380839 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:11.382197 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:11.395758 13292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:11.395893 13289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:11.395911 13290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:11.396914 13284 server_base.cc:1034] running on GCE node
I20260808 15:47:11.397521 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:11.398644 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:11.399890 13284 hybrid_clock.cc:648] HybridClock initialized: now 1786204031399845 us; error 55 us; skew 500 ppm
I20260808 15:47:11.400221 13284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:11 dist-test-slave-gp6n krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:11.403733 13284 init.cc:377] Logged in from keytab as kudu/127.23.130.2@KRBTEST.COM (short username kudu)
I20260808 15:47:11.405329 13284 webserver.cc:460] Webserver started at http://127.23.130.2:39383/ using document root <none> and password file <none>
I20260808 15:47:11.406032 13284 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:11.406185 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:11.406600 13284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:11.409546 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "6d12c779f4d444119d31447f38785136"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "fec2b9a6abbfc7fbfc15a026189926ca"
server_key_iv: "47f9222d8977b8c6e959dec230ace42e"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.410341 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "6d12c779f4d444119d31447f38785136"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "fec2b9a6abbfc7fbfc15a026189926ca"
server_key_iv: "47f9222d8977b8c6e959dec230ace42e"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.415835 13284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 15:47:11.419332 13299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:11.420802 13284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:47:11.421006 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "6d12c779f4d444119d31447f38785136"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "fec2b9a6abbfc7fbfc15a026189926ca"
server_key_iv: "47f9222d8977b8c6e959dec230ace42e"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.421224 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:11.441332 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:11.445803 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:11.446113 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:11.447180 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:11.448563 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:11.448673 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:11.448824 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:11.448904 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:11.484184 13284 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:34779
I20260808 15:47:11.484232 13431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:34779 every 8 connection(s)
I20260808 15:47:11.485504 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:47:11.492952 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13284
I20260808 15:47:11.493155 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:47:11.493618 24072 external_mini_cluster.cc:1428] Setting key d4e8938c8195edd1d63f8a0c32b30ce0
Aug 08 15:47:11 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.2@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:11.508790 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:11.488270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:38689 (local address 127.23.130.62:42857)
0808 15:47:11.488780 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:11.488785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:11.489706 (+ 921us) server_negotiation.cc:407] Connection header received
0808 15:47:11.491679 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:11.491685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:11.491779 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:11.491906 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:11.493632 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:11.494555 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:11.496670 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:11.497451 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:11.497606 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:11.501299 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:11.501308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:11.501316 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:11.501364 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:11.505215 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:11.506001 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:11.506009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:11.506015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:11.506183 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:11.506711 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:11.506719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:11.506725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:11.507117 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:11.507281 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:11.508015 (+ 734us) server_negotiation.cc:299] Negotiation successful
0808 15:47:11.508275 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":281,"threads_started":1}
I20260808 15:47:11.509457 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:11.488761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42857 (local address 127.23.130.2:38689)
0808 15:47:11.489518 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:11.489559 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:11.491366 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:11.492165 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:11.492175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:11.492747 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:11.493447 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:11.493467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:11.494747 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:11.494753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:11.496494 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:11.496508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:11.497706 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:11.497711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:11.497926 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:11.498729 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:11.498759 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:11.501052 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:11.505376 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:11.505384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:11.505397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:11.505868 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:11.506321 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:11.506324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:11.506327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:11.506595 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:11.507258 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:11.507265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:11.507606 (+ 341us) client_negotiation.cc:769] Sending connection context
0808 15:47:11.507866 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 15:47:11.508287 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
I20260808 15:47:11.511112 13432 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42857
I20260808 15:47:11.511443 13432 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:11.512168 13432 heartbeater.cc:511] Master 127.23.130.62:42857 requested a full tablet report, sending...
I20260808 15:47:11.513711 13070 ts_manager.cc:194] Registered new tserver with Master: 6d12c779f4d444119d31447f38785136 (127.23.130.2:34779)
I20260808 15:47:11.514612 13070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.2@KRBTEST.COM'} at 127.23.130.2:38689
WARNING: no policy specified for kudu/127.23.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
Entry for principal kudu/127.23.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab.
I20260808 15:47:11.564214 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:11.904609 13438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:11.904906 13438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:11.905076 13438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:11.912665 13438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:11.912824 13438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:11.913022 13438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:47:11.923010 13438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.3.keytab
--principal=kudu/127.23.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:42857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:11.924329 13438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:11.925807 13438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:11.939550 13443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:11.939678 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:11.939528 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:11.940807 13438 server_base.cc:1034] running on GCE node
I20260808 15:47:11.941397 13438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:11.942303 13438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:11.943534 13438 hybrid_clock.cc:648] HybridClock initialized: now 1786204031943533 us; error 82 us; skew 500 ppm
I20260808 15:47:11.943843 13438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:11 dist-test-slave-gp6n krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:11.947252 13438 init.cc:377] Logged in from keytab as kudu/127.23.130.3@KRBTEST.COM (short username kudu)
I20260808 15:47:11.948772 13438 webserver.cc:460] Webserver started at http://127.23.130.3:33815/ using document root <none> and password file <none>
I20260808 15:47:11.949463 13438 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:11.949610 13438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:11.949928 13438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:11.952899 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "f284041a87fe4087abb2b7dbc6204167"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "a261b4389dfe6c22177d2e8a765a8993"
server_key_iv: "01ba96b3cf24623591c1a03a85f328c7"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.953666 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "f284041a87fe4087abb2b7dbc6204167"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "a261b4389dfe6c22177d2e8a765a8993"
server_key_iv: "01ba96b3cf24623591c1a03a85f328c7"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.958891 13438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:47:11.962276 13455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:11.963723 13438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:47:11.963935 13438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "f284041a87fe4087abb2b7dbc6204167"
format_stamp: "Formatted at 2026-08-08 15:47:11 on dist-test-slave-gp6n"
server_key: "a261b4389dfe6c22177d2e8a765a8993"
server_key_iv: "01ba96b3cf24623591c1a03a85f328c7"
server_key_version: "encryptionkey@0"
I20260808 15:47:11.964154 13438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:11.966779 13277 heartbeater.cc:503] Master 127.23.130.62:42857 was elected leader, sending a full tablet report...
I20260808 15:47:11.989176 13438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:11.994045 13438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:11.994377 13438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:11.995537 13438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:11.996984 13438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:11.997104 13438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:11.997270 13438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:11.997352 13438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:12.033885 13438 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:45827
I20260808 15:47:12.033987 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:45827 every 8 connection(s)
I20260808 15:47:12.035545 13438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:47:12.045631 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13438
I20260808 15:47:12.045821 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:47:12.046317 24072 external_mini_cluster.cc:1428] Setting key 884b9e12b7d446083d5704a05c70a3b9
Aug 08 15:47:12 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.3@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:12.056934 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.038202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:55189 (local address 127.23.130.62:42857)
0808 15:47:12.038815 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.038822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.039800 (+ 978us) server_negotiation.cc:407] Connection header received
0808 15:47:12.041756 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.041760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.041828 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.041930 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:12.043521 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.044324 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.046252 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.046922 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.047066 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.050808 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:12.050814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:12.050819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:12.050859 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:12.053500 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.054525 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.054530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.054533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.054675 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.055130 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.055135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.055138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.055414 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:12.055589 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.056302 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.056534 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":334,"threads_started":1}
I20260808 15:47:12.057881 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.038775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:42857 (local address 127.23.130.3:55189)
0808 15:47:12.039578 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.039617 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.041480 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.042144 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.042152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.042767 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:12.043353 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.043375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.044487 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.044492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.046055 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.046069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.047179 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.047187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.047466 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.048226 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:12.048248 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:12.050593 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:12.053652 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:12.053660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:12.053676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:12.054359 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:12.054796 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:12.054801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:12.054804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:12.055004 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:12.055552 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:12.055560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:12.055922 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.056195 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.056749 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":307,"threads_started":1}
I20260808 15:47:12.059382 13582 heartbeater.cc:347] Connected to a master server at 127.23.130.62:42857
I20260808 15:47:12.059708 13582 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:12.060631 13582 heartbeater.cc:511] Master 127.23.130.62:42857 requested a full tablet report, sending...
I20260808 15:47:12.062317 13070 ts_manager.cc:194] Registered new tserver with Master: f284041a87fe4087abb2b7dbc6204167 (127.23.130.3:45827)
I20260808 15:47:12.063170 13070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.130.3@KRBTEST.COM'} at 127.23.130.3:55189
I20260808 15:47:12.074613 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:47:12.091711 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.078871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.23.130.62:42857)
0808 15:47:12.079147 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.079157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.079563 (+ 406us) server_negotiation.cc:407] Connection header received
0808 15:47:12.079669 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.079673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.079738 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.079822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:12.081320 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.082046 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.083378 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.083779 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.083886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.086348 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:12.086353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:12.086356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:12.086387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:12.089073 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.089746 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.089756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.089759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.089879 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.090303 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.090306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.090308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.090680 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:12.090884 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.091209 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.091373 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:47:12.095623 13070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:47:12.102334 13070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:47:12.129524 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.115471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.23.130.1:40923)
0808 15:47:12.116872 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.116886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.120568 (+ 3682us) server_negotiation.cc:407] Connection header received
0808 15:47:12.120889 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.120900 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.121196 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.122160 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:12.122846 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.123882 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.127305 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.128280 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.128445 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.128703 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.128983 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.129158 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":3014,"threads_started":1}
I20260808 15:47:12.129613 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.114791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40923 (local address 127.0.0.1:53552)
0808 15:47:12.120438 (+ 5647us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.120462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.120671 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.121453 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.121460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.121493 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:12.122650 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.122661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.124018 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.124023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.127125 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.127137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.128507 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.128513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.128708 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.128766 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.128914 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.129138 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5497,"mutex_wait_us":208,"thread_start_us":2674,"threads_started":1}
I20260808 15:47:12.130115 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.114698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34779 (local address 127.0.0.1:44030)
0808 15:47:12.117544 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.117595 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.119101 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.121490 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.121495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.121522 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:12.121993 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.122005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.124150 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.124155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.127085 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.127105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.129117 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.129126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.129354 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.129442 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.129536 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.129686 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"mutex_wait_us":228,"thread_start_us":6139,"threads_started":1}
I20260808 15:47:12.130179 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.117124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.23.130.3:45827)
0808 15:47:12.119893 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.119906 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.121229 (+ 1323us) server_negotiation.cc:407] Connection header received
0808 15:47:12.121404 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.121415 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.121621 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.121802 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:12.122886 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.124371 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.127118 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.128585 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.128724 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.128962 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.129653 (+ 691us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.129830 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2547,"thread_start_us":2359,"threads_started":1}
I20260808 15:47:12.130975 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.114737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.23.130.2:34779)
0808 15:47:12.120830 (+ 6093us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.120842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.120874 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:47:12.120985 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.120993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.121254 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.121682 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:12.122319 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.124014 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.127230 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.128601 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.128724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.128929 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.130458 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.130634 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5894,"thread_start_us":5751,"threads_started":1}
I20260808 15:47:12.131670 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.116412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:45827 (local address 127.0.0.1:44582)
0808 15:47:12.121099 (+ 4687us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.121116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.121241 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.121950 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.121954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.121986 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:12.122631 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.122647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.124586 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.124593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.126933 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.126946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.129167 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.129174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.129370 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.129456 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.129528 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.131319 (+ 1791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4583,"mutex_wait_us":1723,"thread_start_us":3667,"threads_started":1}
I20260808 15:47:12.133910 13206 tablet_service.cc:1467] Processing CreateTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 (DEFAULT_TABLE table=test-table [id=a9bb4427ef7746c398d09210bb892d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:12.134498 13354 tablet_service.cc:1467] Processing CreateTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 (DEFAULT_TABLE table=test-table [id=a9bb4427ef7746c398d09210bb892d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:12.135591 13206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a348e3758cba4d39984eb60aa74ef8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:12.135769 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a348e3758cba4d39984eb60aa74ef8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:12.135637 13508 tablet_service.cc:1467] Processing CreateTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 (DEFAULT_TABLE table=test-table [id=a9bb4427ef7746c398d09210bb892d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:12.136971 13508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a348e3758cba4d39984eb60aa74ef8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:12.147841 13602 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Bootstrap starting.
I20260808 15:47:12.150473 13603 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Bootstrap starting.
I20260808 15:47:12.151957 13602 tablet_bootstrap.cc:654] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:12.153306 13602 log.cc:824] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:12.154235 13603 tablet_bootstrap.cc:654] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:12.155097 13604 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Bootstrap starting.
I20260808 15:47:12.155579 13603 log.cc:824] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:12.156718 13602 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: No bootstrap required, opened a new log
I20260808 15:47:12.157019 13602 ts_tablet_manager.cc:1397] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:47:12.159118 13603 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: No bootstrap required, opened a new log
I20260808 15:47:12.159396 13603 ts_tablet_manager.cc:1397] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:47:12.159422 13604 tablet_bootstrap.cc:654] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:12.160647 13604 log.cc:824] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:12.163728 13604 tablet_bootstrap.cc:492] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: No bootstrap required, opened a new log
I20260808 15:47:12.163976 13604 ts_tablet_manager.cc:1397] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:47:12.164768 13602 raft_consensus.cc:348] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.165194 13602 raft_consensus.cc:374] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:12.165323 13602 raft_consensus.cc:729] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d12c779f4d444119d31447f38785136, State: Initialized, Role: FOLLOWER
I20260808 15:47:12.166074 13602 consensus_queue.cc:260] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.167256 13432 heartbeater.cc:503] Master 127.23.130.62:42857 was elected leader, sending a full tablet report...
I20260808 15:47:12.167937 13602 ts_tablet_manager.cc:1428] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260808 15:47:12.170042 13603 raft_consensus.cc:348] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.170500 13603 raft_consensus.cc:374] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:12.170606 13603 raft_consensus.cc:729] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86327eef1c5e497db26648710e7f5ed9, State: Initialized, Role: FOLLOWER
I20260808 15:47:12.171226 13603 consensus_queue.cc:260] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.172937 13603 ts_tablet_manager.cc:1428] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260808 15:47:12.174552 13604 raft_consensus.cc:348] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.175009 13604 raft_consensus.cc:374] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:12.175123 13604 raft_consensus.cc:729] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f284041a87fe4087abb2b7dbc6204167, State: Initialized, Role: FOLLOWER
I20260808 15:47:12.175647 13604 consensus_queue.cc:260] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.176712 13582 heartbeater.cc:503] Master 127.23.130.62:42857 was elected leader, sending a full tablet report...
I20260808 15:47:12.177439 13604 ts_tablet_manager.cc:1428] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260808 15:47:12.186094 13279 tablet.cc:2369] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:47:12.237620 13433 tablet.cc:2369] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:47:12.287312 13583 tablet.cc:2369] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:12.351488 13610 raft_consensus.cc:482] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:47:12.351825 13610 raft_consensus.cc:504] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.353963 13610 leader_election.cc:288] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d12c779f4d444119d31447f38785136 (127.23.130.2:34779), 86327eef1c5e497db26648710e7f5ed9 (127.23.130.1:40923)
I20260808 15:47:12.361162 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.354613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:49641 (local address 127.23.130.1:40923)
0808 15:47:12.354882 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.354892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.354918 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:47:12.354997 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.355002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.355090 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.355196 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:12.355688 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.356722 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.359159 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.360308 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.360491 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.360568 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.360798 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.360936 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:47:12.361213 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.354264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:40923 (local address 127.23.130.3:49641)
0808 15:47:12.354656 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.354676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.354850 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.355246 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.355251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.355295 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:12.355535 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.355551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.356875 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.356879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.358990 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.359008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.360475 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.360479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.360623 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.360649 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.360734 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.360864 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":52,"spinlock_wait_cycles":7936}
I20260808 15:47:12.361723 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.354319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:59449 (local address 127.23.130.2:34779)
0808 15:47:12.354581 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.354590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.355370 (+ 780us) server_negotiation.cc:407] Connection header received
0808 15:47:12.355552 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.355556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.355646 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.355741 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:12.356257 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.357164 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.359528 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.360754 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.360904 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.360978 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.361370 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.361547 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:47:12.362321 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.354443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:34779 (local address 127.23.130.3:59449)
0808 15:47:12.355235 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:12.355251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:12.355396 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:12.355761 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:12.355764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:12.355784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:12.356075 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.356082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.357340 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.357346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.359366 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:12.359376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.360985 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.360991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:12.361160 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.361192 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:47:12.361909 (+ 717us) client_negotiation.cc:240] Negotiation successful
0808 15:47:12.362052 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":24,"spinlock_wait_cycles":896,"thread_start_us":438,"threads_started":1}
I20260808 15:47:12.362236 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a348e3758cba4d39984eb60aa74ef8f1" candidate_uuid: "f284041a87fe4087abb2b7dbc6204167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86327eef1c5e497db26648710e7f5ed9" is_pre_election: true
I20260808 15:47:12.362849 13227 raft_consensus.cc:2454] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f284041a87fe4087abb2b7dbc6204167 in term 0.
I20260808 15:47:12.363397 13374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a348e3758cba4d39984eb60aa74ef8f1" candidate_uuid: "f284041a87fe4087abb2b7dbc6204167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d12c779f4d444119d31447f38785136" is_pre_election: true
I20260808 15:47:12.363608 13459 leader_election.cc:302] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86327eef1c5e497db26648710e7f5ed9, f284041a87fe4087abb2b7dbc6204167; no voters:
I20260808 15:47:12.363920 13374 raft_consensus.cc:2454] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f284041a87fe4087abb2b7dbc6204167 in term 0.
I20260808 15:47:12.364025 13610 raft_consensus.cc:2781] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:47:12.364197 13610 raft_consensus.cc:482] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:47:12.364312 13610 raft_consensus.cc:3037] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:12.366664 13610 raft_consensus.cc:504] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.367280 13610 leader_election.cc:288] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [CANDIDATE]: Term 1 election: Requested vote from peers 6d12c779f4d444119d31447f38785136 (127.23.130.2:34779), 86327eef1c5e497db26648710e7f5ed9 (127.23.130.1:40923)
I20260808 15:47:12.367676 13374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a348e3758cba4d39984eb60aa74ef8f1" candidate_uuid: "f284041a87fe4087abb2b7dbc6204167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d12c779f4d444119d31447f38785136"
I20260808 15:47:12.367790 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a348e3758cba4d39984eb60aa74ef8f1" candidate_uuid: "f284041a87fe4087abb2b7dbc6204167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86327eef1c5e497db26648710e7f5ed9"
I20260808 15:47:12.367887 13374 raft_consensus.cc:3037] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:12.367995 13227 raft_consensus.cc:3037] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:12.371201 13374 raft_consensus.cc:2454] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f284041a87fe4087abb2b7dbc6204167 in term 1.
I20260808 15:47:12.371201 13227 raft_consensus.cc:2454] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f284041a87fe4087abb2b7dbc6204167 in term 1.
I20260808 15:47:12.371584 13459 leader_election.cc:302] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86327eef1c5e497db26648710e7f5ed9, f284041a87fe4087abb2b7dbc6204167; no voters:
I20260808 15:47:12.371980 13610 raft_consensus.cc:2781] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:12.372398 13610 raft_consensus.cc:686] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 1 LEADER]: Becoming Leader. State: Replica: f284041a87fe4087abb2b7dbc6204167, State: Running, Role: LEADER
I20260808 15:47:12.372874 13610 consensus_queue.cc:237] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } }
I20260808 15:47:12.377518 13070 catalog_manager.cc:5495] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 reported cstate change: term changed from 0 to 1, leader changed from <none> to f284041a87fe4087abb2b7dbc6204167 (127.23.130.3). New cstate: current_term: 1 leader_uuid: "f284041a87fe4087abb2b7dbc6204167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f284041a87fe4087abb2b7dbc6204167" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 45827 } health_report { overall_health: HEALTHY } } }
I20260808 15:47:12.422945 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.417440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.23.130.3:45827)
0808 15:47:12.417685 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.417691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.417710 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:47:12.417912 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.417916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.417993 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.418150 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:12.418604 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.419343 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.420780 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.421177 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.421316 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.421822 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:12.422321 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:12.422511 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.422592 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.422670 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:47:12.431902 13374 raft_consensus.cc:1261] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 1 FOLLOWER]: Refusing update from remote peer f284041a87fe4087abb2b7dbc6204167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:12.431905 13227 raft_consensus.cc:1261] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 1 FOLLOWER]: Refusing update from remote peer f284041a87fe4087abb2b7dbc6204167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:12.433106 13613 consensus_queue.cc:1035] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86327eef1c5e497db26648710e7f5ed9" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:12.433598 13610 consensus_queue.cc:1035] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d12c779f4d444119d31447f38785136" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:12.464682 13068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:12.465021 13068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:12.468782 13068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddc41dc2cbb14b3fb07320d1bb437bd8: Sending DeleteTablet for 3 replicas of tablet a348e3758cba4d39984eb60aa74ef8f1
I20260808 15:47:12.470021 13354 tablet_service.cc:1514] Processing DeleteTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:12 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:44030
I20260808 15:47:12.470160 13206 tablet_service.cc:1514] Processing DeleteTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:12 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:53552
I20260808 15:47:12.470886 13508 tablet_service.cc:1514] Processing DeleteTablet for tablet a348e3758cba4d39984eb60aa74ef8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:12 UTC) from {username='kudu', principal='kudu/127.23.130.62@KRBTEST.COM'} at 127.0.0.1:44582
I20260808 15:47:12.472997 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13137
I20260808 15:47:12.499892 13628 tablet_replica.cc:330] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: stopping tablet replica
I20260808 15:47:12.500546 13627 tablet_replica.cc:330] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: stopping tablet replica
I20260808 15:47:12.500511 13628 raft_consensus.cc:2229] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:47:12.501044 13627 raft_consensus.cc:2229] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:12.501446 13628 raft_consensus.cc:2258] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:12.501605 13627 raft_consensus.cc:2258] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:12.504175 13626 tablet_replica.cc:330] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: stopping tablet replica
I20260808 15:47:12.504709 13626 raft_consensus.cc:2229] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:12.505295 13626 raft_consensus.cc:2258] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:12.507750 13628 ts_tablet_manager.cc:1905] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:12.508531 13626 ts_tablet_manager.cc:1905] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:47:12 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.1@KRBTEST.COM
I20260808 15:47:12.512795 13627 ts_tablet_manager.cc:1905] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:12.516408 13626 ts_tablet_manager.cc:1918] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:12.516585 13626 log.cc:1196] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/a348e3758cba4d39984eb60aa74ef8f1
I20260808 15:47:12.517577 13626 ts_tablet_manager.cc:1939] T a348e3758cba4d39984eb60aa74ef8f1 P 6d12c779f4d444119d31447f38785136: Deleting consensus metadata
I20260808 15:47:12.519495 13628 ts_tablet_manager.cc:1918] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:12.519675 13628 log.cc:1196] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/a348e3758cba4d39984eb60aa74ef8f1
I20260808 15:47:12.519974 13050 catalog_manager.cc:4841] TS 6d12c779f4d444119d31447f38785136 (127.23.130.2:34779): tablet a348e3758cba4d39984eb60aa74ef8f1 (table test-table [id=a9bb4427ef7746c398d09210bb892d65]) successfully deleted
I20260808 15:47:12.520351 13628 ts_tablet_manager.cc:1939] T a348e3758cba4d39984eb60aa74ef8f1 P f284041a87fe4087abb2b7dbc6204167: Deleting consensus metadata
I20260808 15:47:12.522228 13051 catalog_manager.cc:4841] TS f284041a87fe4087abb2b7dbc6204167 (127.23.130.3:45827): tablet a348e3758cba4d39984eb60aa74ef8f1 (table test-table [id=a9bb4427ef7746c398d09210bb892d65]) successfully deleted
I20260808 15:47:12.523761 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.474976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.23.130.1:40923)
0808 15:47:12.475217 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.475227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.495919 (+ 20692us) server_negotiation.cc:407] Connection header received
0808 15:47:12.496569 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.496578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.496723 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.496821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:12.501949 (+ 5128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.503567 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.506114 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.506821 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.506933 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.515810 (+ 8877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:12.515823 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:12.515839 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:12.515890 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:12.519957 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.520874 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.520884 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.520887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.521070 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.521568 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.521573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.521576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.522016 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:12.522302 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.523006 (+ 704us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.523285 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:47:12.523850 13627 ts_tablet_manager.cc:1918] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:12.524230 13627 log.cc:1196] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/a348e3758cba4d39984eb60aa74ef8f1
I20260808 15:47:12.524821 13627 ts_tablet_manager.cc:1939] T a348e3758cba4d39984eb60aa74ef8f1 P 86327eef1c5e497db26648710e7f5ed9: Deleting consensus metadata
I20260808 15:47:12.524847 13272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:12.615412 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13137
I20260808 15:47:12.615530 13053 catalog_manager.cc:4841] TS 86327eef1c5e497db26648710e7f5ed9 (127.23.130.1:40923): tablet a348e3758cba4d39984eb60aa74ef8f1 (table test-table [id=a9bb4427ef7746c398d09210bb892d65]) successfully deleted
I20260808 15:47:12.644104 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13284
Aug 08 15:47:12 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.2@KRBTEST.COM
I20260808 15:47:12.662297 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.645048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.23.130.2:34779)
0808 15:47:12.645256 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.645264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.645292 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:47:12.645375 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.645381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.645472 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.645606 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:12.646986 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.648364 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.649652 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.650373 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.650539 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.654961 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:12.654980 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:12.655001 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:12.655049 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:12.659436 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.660086 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.660097 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.660101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.660266 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.660619 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.660625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.660629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.661144 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:12.661482 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.661645 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.661896 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:47:12.663105 13426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:12.755889 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13284
I20260808 15:47:12.777096 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13438
Aug 08 15:47:12 dist-test-slave-gp6n krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.3@KRBTEST.COM
I20260808 15:47:12.796126 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:12.777910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.23.130.3:45827)
0808 15:47:12.778093 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:12.778098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:12.778206 (+ 108us) server_negotiation.cc:407] Connection header received
0808 15:47:12.778454 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:12.778466 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:12.778553 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:12.778648 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:12.780539 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.781358 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.783336 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:12.783837 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:12.783934 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:12.789708 (+ 5774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:12.789720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:12.789738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:12.789770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:12.792891 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.793623 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.793640 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.793642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.793800 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:12.794213 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:12.794218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:12.794221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:12.794874 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:12.795202 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:12.795512 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 15:47:12.795733 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:47:12.796784 13576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:12.921295 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13438
I20260808 15:47:12.942893 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13034
I20260808 15:47:12.944077 13105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:13.059060 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13034
2026-08-08T15:47:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4817 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T15:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:13Z Disabled control of system clock
I20260808 15:47:13.155357 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33275
--webserver_interface=127.23.130.62
--webserver_port=0
--builtin_ntp_servers=127.23.130.20:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:47:13.507223 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:13.507512 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:13.507714 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:13.515115 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:13.515216 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:13.515307 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:13.515414 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:13.515498 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:13.524657 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:33275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:13.525909 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:13.527746 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:13.540273 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:13.540359 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:13.540302 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:13.541411 13637 server_base.cc:1034] running on GCE node
I20260808 15:47:13.542294 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:13.543741 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:13.544992 13637 hybrid_clock.cc:648] HybridClock initialized: now 1786204033544926 us; error 56 us; skew 500 ppm
I20260808 15:47:13.545353 13637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:13.552258 13637 webserver.cc:460] Webserver started at http://127.23.130.62:37321/ using document root <none> and password file <none>
I20260808 15:47:13.553006 13637 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:13.553148 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:13.553484 13637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:13.556474 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "2f02702c3cf6487cbd183212b7ff02b7"
format_stamp: "Formatted at 2026-08-08 15:47:13 on dist-test-slave-gp6n"
server_key: "ca1c0d8559203bae0a3b55ce912fd271"
server_key_iv: "4f440cd887153ccae91605d45c253ff9"
server_key_version: "encryptionkey@0"
I20260808 15:47:13.557322 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "2f02702c3cf6487cbd183212b7ff02b7"
format_stamp: "Formatted at 2026-08-08 15:47:13 on dist-test-slave-gp6n"
server_key: "ca1c0d8559203bae0a3b55ce912fd271"
server_key_iv: "4f440cd887153ccae91605d45c253ff9"
server_key_version: "encryptionkey@0"
I20260808 15:47:13.562378 13637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:47:13.565708 13654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:13.567092 13637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 15:47:13.567281 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "2f02702c3cf6487cbd183212b7ff02b7"
format_stamp: "Formatted at 2026-08-08 15:47:13 on dist-test-slave-gp6n"
server_key: "ca1c0d8559203bae0a3b55ce912fd271"
server_key_iv: "4f440cd887153ccae91605d45c253ff9"
server_key_version: "encryptionkey@0"
I20260808 15:47:13.567490 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:13.587311 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:13.588155 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:13.588454 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:13.606992 13637 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:33275
I20260808 15:47:13.607031 13717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:33275 every 8 connection(s)
I20260808 15:47:13.608289 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:13.612018 13718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:13.616020 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13637
I20260808 15:47:13.616215 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:13.616847 24072 external_mini_cluster.cc:1428] Setting key e03627af730a118420117fe4bb05f85b
I20260808 15:47:13.623333 13718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: Bootstrap starting.
I20260808 15:47:13.627080 13718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:13.628397 13718 log.cc:824] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:13.631790 13721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:13.618807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.23.130.62:33275)
0808 15:47:13.619811 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:13.619822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:13.619872 (+ 50us) server_negotiation.cc:407] Connection header received
0808 15:47:13.621697 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:13.621720 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:13.622156 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:13.622536 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:13.623189 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:13.624600 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:13.626630 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:13.627469 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:13.627633 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:13.628923 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:13.628932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:13.628946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:13.629562 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:13.629646 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:13.629667 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:13.629766 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:13.630111 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 15:47:13.630527 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":357,"threads_started":1}
I20260808 15:47:13.632140 13718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: No bootstrap required, opened a new log
I20260808 15:47:13.640141 13718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } }
I20260808 15:47:13.640488 13718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:13.640569 13718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f02702c3cf6487cbd183212b7ff02b7, State: Initialized, Role: FOLLOWER
I20260808 15:47:13.641060 13718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } }
I20260808 15:47:13.641260 13718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:13.641363 13718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:13.641492 13718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:13.643725 13718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } }
I20260808 15:47:13.644192 13718 leader_election.cc:302] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f02702c3cf6487cbd183212b7ff02b7; no voters:
I20260808 15:47:13.644776 13718 leader_election.cc:288] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:13.644946 13723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:13.646942 13718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:13.649015 13723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2f02702c3cf6487cbd183212b7ff02b7, State: Running, Role: LEADER
I20260808 15:47:13.649757 13723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } }
I20260808 15:47:13.653111 13725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f02702c3cf6487cbd183212b7ff02b7. Latest consensus state: current_term: 1 leader_uuid: "2f02702c3cf6487cbd183212b7ff02b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } } }
I20260808 15:47:13.653684 13725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:13.654847 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f02702c3cf6487cbd183212b7ff02b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f02702c3cf6487cbd183212b7ff02b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33275 } } }
I20260808 15:47:13.655237 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7 [sys.catalog]: This master's current role is: LEADER
W20260808 15:47:13.663406 13738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:47:13.663554 13738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:47:13.663717 13739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:13.665370 13739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:13.673468 13739 catalog_manager.cc:1347] Generated new cluster ID: 34ea1a0e15b1443a889c109fa64e50e6
I20260808 15:47:13.673581 13739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:13.686961 13739 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:13.688802 13739 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:13.700446 13739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: Generated new TSK 0
I20260808 15:47:13.701539 13739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:13.709067 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.1:0
--local_ip_for_outbound_sockets=127.23.130.1
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--builtin_ntp_servers=127.23.130.20:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:47:14.058079 13744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:14.058364 13744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:14.058528 13744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:14.065793 13744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:14.065888 13744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:14.065992 13744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.1
I20260808 15:47:14.075181 13744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.130.1
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.1
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:14.076406 13744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:14.078092 13744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:14.091094 13752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:14.091316 13749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:14.091153 13750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:14.092252 13744 server_base.cc:1034] running on GCE node
I20260808 15:47:14.092840 13744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:14.093753 13744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:14.094990 13744 hybrid_clock.cc:648] HybridClock initialized: now 1786204034094943 us; error 57 us; skew 500 ppm
I20260808 15:47:14.095338 13744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:14.097572 13744 webserver.cc:460] Webserver started at http://127.23.130.1:33981/ using document root <none> and password file <none>
I20260808 15:47:14.098237 13744 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:14.098374 13744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:14.098708 13744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:14.101528 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data/instance:
uuid: "4e9c2342924e4848a50e44b928c798ec"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "9a15b575d78d464b577587b191a1a02b"
server_key_iv: "ec604bdd476da8db97c5e2ffa9ce9047"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.102317 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance:
uuid: "4e9c2342924e4848a50e44b928c798ec"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "9a15b575d78d464b577587b191a1a02b"
server_key_iv: "ec604bdd476da8db97c5e2ffa9ce9047"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.107522 13744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:47:14.111017 13758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.112353 13744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:47:14.112521 13744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal
uuid: "4e9c2342924e4848a50e44b928c798ec"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "9a15b575d78d464b577587b191a1a02b"
server_key_iv: "ec604bdd476da8db97c5e2ffa9ce9047"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.112735 13744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:14.138594 13744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:14.139472 13744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:14.139787 13744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:14.140837 13744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:14.142271 13744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:14.142411 13744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.142580 13744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:14.142675 13744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.177423 13744 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.1:39669
I20260808 15:47:14.177515 13882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.1:39669 every 8 connection(s)
I20260808 15:47:14.178848 13744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/data/info.pb
I20260808 15:47:14.181020 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13744
I20260808 15:47:14.181222 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal/instance
I20260808 15:47:14.181771 24072 external_mini_cluster.cc:1428] Setting key b03f9f5ffda76c617d5fad9bbb8b8a01
I20260808 15:47:14.184597 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.2:0
--local_ip_for_outbound_sockets=127.23.130.2
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--builtin_ntp_servers=127.23.130.20:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:47:14.192901 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:14.181596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.1:34417 (local address 127.23.130.62:33275)
0808 15:47:14.182145 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:14.182151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:14.183500 (+ 1349us) server_negotiation.cc:407] Connection header received
0808 15:47:14.185450 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:14.185458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:14.185557 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:14.185683 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:14.186828 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.187980 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.189563 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.190268 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.190447 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:14.191850 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:14.191859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:14.191866 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:14.191892 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:14.191949 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:14.191968 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:14.192066 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:14.192340 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 15:47:14.192453 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
I20260808 15:47:14.193701 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:14.182215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33275 (local address 127.23.130.1:34417)
0808 15:47:14.183338 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:14.183386 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:14.185171 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:14.185869 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:14.185877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:14.185971 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:14.186660 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:14.186680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.188150 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.188154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:14.189395 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:14.189408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.190565 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.190571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:14.190789 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:14.191545 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:14.191548 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:14.191586 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:14.191664 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:14.192094 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:14.192105 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:14.192112 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:47:14.192313 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 15:47:14.192608 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":398,"threads_started":1}
I20260808 15:47:14.195256 13883 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33275
I20260808 15:47:14.195600 13883 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:14.196338 13883 heartbeater.cc:511] Master 127.23.130.62:33275 requested a full tablet report, sending...
I20260808 15:47:14.198335 13671 ts_manager.cc:194] Registered new tserver with Master: 4e9c2342924e4848a50e44b928c798ec (127.23.130.1:39669)
I20260808 15:47:14.200641 13671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.1:34417
W20260808 15:47:14.530301 13888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:14.530608 13888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:14.530697 13888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:14.537900 13888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:14.538000 13888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:14.538110 13888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.2
I20260808 15:47:14.547106 13888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.130.2
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.2
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:14.548626 13888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:14.550509 13888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:14.563658 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:14.563755 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:14.563731 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:14.565173 13888 server_base.cc:1034] running on GCE node
I20260808 15:47:14.565730 13888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:14.566651 13888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:14.567890 13888 hybrid_clock.cc:648] HybridClock initialized: now 1786204034567835 us; error 44 us; skew 500 ppm
I20260808 15:47:14.568220 13888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:14.570523 13888 webserver.cc:460] Webserver started at http://127.23.130.2:44769/ using document root <none> and password file <none>
I20260808 15:47:14.571305 13888 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:14.571434 13888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:14.571827 13888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:14.574806 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data/instance:
uuid: "96e59e03157546c09ec3f7e3e32a8c14"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "6124b8bd5f2f4f4b9a872c3d569a75bc"
server_key_iv: "56a124f2aa30c273701b1fedcfb0e328"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.575651 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance:
uuid: "96e59e03157546c09ec3f7e3e32a8c14"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "6124b8bd5f2f4f4b9a872c3d569a75bc"
server_key_iv: "56a124f2aa30c273701b1fedcfb0e328"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.581087 13888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:14.584692 13904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.586023 13888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:47:14.586199 13888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal
uuid: "96e59e03157546c09ec3f7e3e32a8c14"
format_stamp: "Formatted at 2026-08-08 15:47:14 on dist-test-slave-gp6n"
server_key: "6124b8bd5f2f4f4b9a872c3d569a75bc"
server_key_iv: "56a124f2aa30c273701b1fedcfb0e328"
server_key_version: "encryptionkey@0"
I20260808 15:47:14.586467 13888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:14.600726 13888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:14.601537 13888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:14.601850 13888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:14.603012 13888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:14.604483 13888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:14.604615 13888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.604758 13888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:14.604848 13888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:14.640360 13888 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.2:33169
I20260808 15:47:14.640421 14032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.2:33169 every 8 connection(s)
I20260808 15:47:14.641669 13888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/data/info.pb
I20260808 15:47:14.648977 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 13888
I20260808 15:47:14.649181 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal/instance
I20260808 15:47:14.649753 24072 external_mini_cluster.cc:1428] Setting key 4b0e929775056561b0ad06177cb05f96
I20260808 15:47:14.652774 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.130.3:0
--local_ip_for_outbound_sockets=127.23.130.3
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--builtin_ntp_servers=127.23.130.20:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:47:14.654897 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:14.644314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.2:32947 (local address 127.23.130.62:33275)
0808 15:47:14.644498 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:14.644503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:14.645879 (+ 1376us) server_negotiation.cc:407] Connection header received
0808 15:47:14.647824 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:14.647831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:14.647924 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:14.648035 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:14.649368 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.650237 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.652065 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.652443 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.652546 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:14.654165 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:14.654170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:14.654174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:14.654191 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:14.654222 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:14.654234 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:14.654305 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:14.654596 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:47:14.654674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:47:14.656356 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:14.644814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33275 (local address 127.23.130.2:32947)
0808 15:47:14.645699 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:14.645738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:14.647522 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:14.648270 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:14.648277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:14.648364 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:14.649171 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:14.649200 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.650466 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.650469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:14.651914 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:14.651930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:14.652687 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:14.652695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:14.653070 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:14.653805 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:14.653811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:14.653875 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:14.653976 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:14.654334 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:14.654348 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:14.654359 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:47:14.654575 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 15:47:14.654998 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":291,"threads_started":1}
I20260808 15:47:14.657825 14033 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33275
I20260808 15:47:14.658138 14033 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:14.659073 14033 heartbeater.cc:511] Master 127.23.130.62:33275 requested a full tablet report, sending...
I20260808 15:47:14.660512 13671 ts_manager.cc:194] Registered new tserver with Master: 96e59e03157546c09ec3f7e3e32a8c14 (127.23.130.2:33169)
I20260808 15:47:14.661314 13671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.2:32947
W20260808 15:47:14.996451 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:14.996737 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:14.996868 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:15.004169 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:15.004272 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:15.004407 14038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.130.3
I20260808 15:47:15.013593 14038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.130.3
--webserver_port=0
--tserver_master_addrs=127.23.130.62:33275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.130.3
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:15.014971 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:15.016844 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:15.030529 14049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:15.030918 14045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:15.030920 14046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:15.031948 14038 server_base.cc:1034] running on GCE node
I20260808 15:47:15.032598 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:15.033741 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:15.034943 14038 hybrid_clock.cc:648] HybridClock initialized: now 1786204035034878 us; error 57 us; skew 500 ppm
I20260808 15:47:15.035279 14038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:47:15.037657 14038 webserver.cc:460] Webserver started at http://127.23.130.3:37157/ using document root <none> and password file <none>
I20260808 15:47:15.038376 14038 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:15.038563 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:15.038908 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:15.041916 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data/instance:
uuid: "662f0d33c9814c9d8e552037e9e238dc"
format_stamp: "Formatted at 2026-08-08 15:47:15 on dist-test-slave-gp6n"
server_key: "ba7b862a5406c42bff56d1ac2f614f7c"
server_key_iv: "31c4eb4dc13ffda65031cd73ef1ab272"
server_key_version: "encryptionkey@0"
I20260808 15:47:15.042834 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance:
uuid: "662f0d33c9814c9d8e552037e9e238dc"
format_stamp: "Formatted at 2026-08-08 15:47:15 on dist-test-slave-gp6n"
server_key: "ba7b862a5406c42bff56d1ac2f614f7c"
server_key_iv: "31c4eb4dc13ffda65031cd73ef1ab272"
server_key_version: "encryptionkey@0"
I20260808 15:47:15.048249 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:15.051733 14056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:15.053006 14038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:47:15.053181 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal
uuid: "662f0d33c9814c9d8e552037e9e238dc"
format_stamp: "Formatted at 2026-08-08 15:47:15 on dist-test-slave-gp6n"
server_key: "ba7b862a5406c42bff56d1ac2f614f7c"
server_key_iv: "31c4eb4dc13ffda65031cd73ef1ab272"
server_key_version: "encryptionkey@0"
I20260808 15:47:15.053392 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:15.077559 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:15.078459 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:15.078776 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:15.079807 14038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:47:15.081313 14038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:47:15.081401 14038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:15.081568 14038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:47:15.081663 14038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:15.115410 14038 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.3:34559
I20260808 15:47:15.115600 14182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.3:34559 every 8 connection(s)
I20260808 15:47:15.116858 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/data/info.pb
I20260808 15:47:15.126273 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14038
I20260808 15:47:15.126489 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal/instance
I20260808 15:47:15.126940 24072 external_mini_cluster.cc:1428] Setting key 9051ac007e2cee01d57cfb86054b6556
I20260808 15:47:15.129966 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.119684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:35253 (local address 127.23.130.62:33275)
0808 15:47:15.119876 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.119880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.121201 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 15:47:15.123067 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.123074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.123170 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.123284 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:15.124370 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.125139 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.126879 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.127397 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.127538 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.129117 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:15.129124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:15.129128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:15.129148 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:15.129185 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:15.129201 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:15.129314 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.129511 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.129611 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:47:15.130968 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.120145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:33275 (local address 127.23.130.3:35253)
0808 15:47:15.121039 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.121083 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.122788 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.123518 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.123526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.123616 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:15.124195 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.124211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.125310 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.125314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.126729 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.126741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.127644 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.127653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.127964 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.128726 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:47:15.128729 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:15.128784 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:47:15.128889 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:15.129316 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:15.129324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:15.129329 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.129474 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.129834 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260808 15:47:15.132237 14183 heartbeater.cc:347] Connected to a master server at 127.23.130.62:33275
I20260808 15:47:15.132547 14183 heartbeater.cc:464] Registering TS with master...
I20260808 15:47:15.133255 14183 heartbeater.cc:511] Master 127.23.130.62:33275 requested a full tablet report, sending...
I20260808 15:47:15.134729 13671 ts_manager.cc:194] Registered new tserver with Master: 662f0d33c9814c9d8e552037e9e238dc (127.23.130.3:34559)
I20260808 15:47:15.135473 13671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.130.3:35253
I20260808 15:47:15.142184 24072 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:47:15.152060 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.146191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.23.130.62:33275)
0808 15:47:15.146490 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.146500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.146872 (+ 372us) server_negotiation.cc:407] Connection header received
0808 15:47:15.146996 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.147000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.147070 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.147175 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:15.147615 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.148401 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.149846 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.150232 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.150382 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.151389 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:15.151394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:15.151397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:15.151414 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:15.151446 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:15.151466 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:15.151535 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.151657 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.151731 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:47:15.155628 13671 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:47:15.162568 13671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:47:15.200528 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.173991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33169 (local address 127.0.0.1:42372)
0808 15:47:15.185180 (+ 11189us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.185259 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.185483 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.192844 (+ 7361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.192852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.192885 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:15.193566 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.193581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.195346 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.195354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.197518 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.197539 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.199565 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.199571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.199747 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.199795 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.199961 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.200151 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10827,"mutex_wait_us":146,"thread_start_us":18419,"threads_started":1}
I20260808 15:47:15.200840 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.175618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.23.130.3:34559)
0808 15:47:15.186278 (+ 10660us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.186291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.192819 (+ 6528us) server_negotiation.cc:407] Connection header received
0808 15:47:15.193120 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.193128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.193399 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.193599 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:15.194362 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.195973 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.198222 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.199701 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.199838 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.200152 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.200277 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.200455 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10430,"thread_start_us":10206,"threads_started":1}
I20260808 15:47:15.201380 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.174210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.23.130.2:33169)
0808 15:47:15.192281 (+ 18071us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.192290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.192314 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:47:15.192400 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.192406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.192593 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.193216 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:15.193823 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.195216 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.197879 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.199365 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.199536 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.199779 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.200840 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.201009 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17922,"thread_start_us":17801,"threads_started":1}
I20260808 15:47:15.201480 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.174257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:39669 (local address 127.0.0.1:50518)
0808 15:47:15.190198 (+ 15941us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.190223 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.190374 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.193045 (+ 2671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.193050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.193077 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:15.193552 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.193564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.196217 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.196222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.198528 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.198542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.200690 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.200696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.200892 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.200933 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.201028 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.201163 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15742,"mutex_wait_us":26,"thread_start_us":16262,"threads_started":1}
I20260808 15:47:15.200712 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.174816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.3:34559 (local address 127.0.0.1:33616)
0808 15:47:15.192682 (+ 17866us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.192702 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.192929 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.193748 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.193752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.193776 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:15.194142 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.194156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.196175 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.196180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.197980 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.197994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.199925 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.199934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.200118 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.200149 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.200233 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.200343 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17740,"mutex_wait_us":88,"thread_start_us":9790,"threads_started":1}
I20260808 15:47:15.202333 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.174854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.23.130.1:39669)
0808 15:47:15.192136 (+ 17282us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.192145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.192177 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:47:15.192281 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.192289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.192537 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.192751 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:15.193732 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.196073 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.198744 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.200153 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.200284 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.200545 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.201846 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.202018 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17099,"thread_start_us":17749,"threads_started":1}
I20260808 15:47:15.203962 13883 heartbeater.cc:503] Master 127.23.130.62:33275 was elected leader, sending a full tablet report...
I20260808 15:47:15.205010 14111 tablet_service.cc:1467] Processing CreateTablet for tablet f3fd690d030a4ee7b671e96a68af3527 (DEFAULT_TABLE table=test-table [id=d368a694c3714a51aa484043f37df369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:15.205008 13956 tablet_service.cc:1467] Processing CreateTablet for tablet f3fd690d030a4ee7b671e96a68af3527 (DEFAULT_TABLE table=test-table [id=d368a694c3714a51aa484043f37df369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:15.206364 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fd690d030a4ee7b671e96a68af3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:15.206423 14111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fd690d030a4ee7b671e96a68af3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:15.209363 13815 tablet_service.cc:1467] Processing CreateTablet for tablet f3fd690d030a4ee7b671e96a68af3527 (DEFAULT_TABLE table=test-table [id=d368a694c3714a51aa484043f37df369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:47:15.210776 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fd690d030a4ee7b671e96a68af3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:15.218482 14204 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Bootstrap starting.
I20260808 15:47:15.222141 14204 tablet_bootstrap.cc:654] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:15.223258 14204 log.cc:824] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:15.226999 14207 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Bootstrap starting.
I20260808 15:47:15.228729 14205 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Bootstrap starting.
I20260808 15:47:15.232014 14205 tablet_bootstrap.cc:654] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:15.232256 14207 tablet_bootstrap.cc:654] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:15.232568 14204 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: No bootstrap required, opened a new log
I20260808 15:47:15.232868 14204 ts_tablet_manager.cc:1397] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260808 15:47:15.233186 14205 log.cc:824] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:15.233512 14207 log.cc:824] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:15.236037 14207 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: No bootstrap required, opened a new log
I20260808 15:47:15.236304 14207 ts_tablet_manager.cc:1397] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260808 15:47:15.236357 14205 tablet_bootstrap.cc:492] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: No bootstrap required, opened a new log
I20260808 15:47:15.236626 14205 ts_tablet_manager.cc:1397] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:47:15.243712 14207 raft_consensus.cc:348] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.243687 14204 raft_consensus.cc:348] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.244119 14207 raft_consensus.cc:374] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:15.244135 14204 raft_consensus.cc:374] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:15.244225 14204 raft_consensus.cc:729] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662f0d33c9814c9d8e552037e9e238dc, State: Initialized, Role: FOLLOWER
I20260808 15:47:15.244270 14207 raft_consensus.cc:729] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9c2342924e4848a50e44b928c798ec, State: Initialized, Role: FOLLOWER
I20260808 15:47:15.244869 14207 consensus_queue.cc:260] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.244881 14204 consensus_queue.cc:260] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.246624 14207 ts_tablet_manager.cc:1428] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260808 15:47:15.248246 14183 heartbeater.cc:503] Master 127.23.130.62:33275 was elected leader, sending a full tablet report...
I20260808 15:47:15.247913 14205 raft_consensus.cc:348] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.248409 14205 raft_consensus.cc:374] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:15.248503 14205 raft_consensus.cc:729] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e59e03157546c09ec3f7e3e32a8c14, State: Initialized, Role: FOLLOWER
I20260808 15:47:15.248975 14204 ts_tablet_manager.cc:1428] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260808 15:47:15.249075 14205 consensus_queue.cc:260] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.250352 14033 heartbeater.cc:503] Master 127.23.130.62:33275 was elected leader, sending a full tablet report...
I20260808 15:47:15.251060 14205 ts_tablet_manager.cc:1428] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260808 15:47:15.368626 14184 tablet.cc:2369] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:15.379824 14211 raft_consensus.cc:482] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:47:15.380198 14211 raft_consensus.cc:504] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.381896 14211 leader_election.cc:288] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e59e03157546c09ec3f7e3e32a8c14 (127.23.130.2:33169), 4e9c2342924e4848a50e44b928c798ec (127.23.130.1:39669)
I20260808 15:47:15.388501 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.382259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:34889 (local address 127.23.130.2:33169)
0808 15:47:15.382497 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.382504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.382981 (+ 477us) server_negotiation.cc:407] Connection header received
0808 15:47:15.383067 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.383073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.383164 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.383282 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:15.383858 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.384740 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.386731 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.387561 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.387679 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.387728 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.388095 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.388233 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:47:15.388659 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.382706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.130.3:46947 (local address 127.23.130.1:39669)
0808 15:47:15.382937 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.382943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.383278 (+ 335us) server_negotiation.cc:407] Connection header received
0808 15:47:15.383450 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.383458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.383543 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.383651 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:47:15.384275 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.385049 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.386939 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.387792 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.387945 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.388028 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.388290 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.388412 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:47:15.388506 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.382262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.2:33169 (local address 127.23.130.3:34889)
0808 15:47:15.382631 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.382647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.382813 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.383418 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.383423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.383472 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:15.383672 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.383683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.384891 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.384897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.386576 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.386588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.387722 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.387731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.387917 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.387959 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.388054 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.388165 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":86}
I20260808 15:47:15.388757 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.382491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.1:39669 (local address 127.23.130.3:46947)
0808 15:47:15.383170 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:15.383188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:15.383309 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:15.383669 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:15.383672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:15.383691 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:47:15.384101 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.384115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.385202 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.385206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.386746 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:15.386760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.387954 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.387959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:15.388120 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.388151 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:47:15.388241 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 15:47:15.388359 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":21,"thread_start_us":305,"threads_started":1}
I20260808 15:47:15.389976 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fd690d030a4ee7b671e96a68af3527" candidate_uuid: "662f0d33c9814c9d8e552037e9e238dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e59e03157546c09ec3f7e3e32a8c14" is_pre_election: true
I20260808 15:47:15.390000 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fd690d030a4ee7b671e96a68af3527" candidate_uuid: "662f0d33c9814c9d8e552037e9e238dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9c2342924e4848a50e44b928c798ec" is_pre_election: true
I20260808 15:47:15.390532 13980 raft_consensus.cc:2454] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662f0d33c9814c9d8e552037e9e238dc in term 0.
I20260808 15:47:15.390554 13835 raft_consensus.cc:2454] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662f0d33c9814c9d8e552037e9e238dc in term 0.
I20260808 15:47:15.391279 14061 leader_election.cc:302] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e9c2342924e4848a50e44b928c798ec, 662f0d33c9814c9d8e552037e9e238dc, 96e59e03157546c09ec3f7e3e32a8c14; no voters:
I20260808 15:47:15.391685 14211 raft_consensus.cc:2781] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:47:15.391836 14211 raft_consensus.cc:482] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:47:15.391969 14211 raft_consensus.cc:3037] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 0 FOLLOWER]: Advancing to term 1
W20260808 15:47:15.393666 14034 tablet.cc:2369] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:15.394251 14211 raft_consensus.cc:504] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.394922 14211 leader_election.cc:288] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [CANDIDATE]: Term 1 election: Requested vote from peers 96e59e03157546c09ec3f7e3e32a8c14 (127.23.130.2:33169), 4e9c2342924e4848a50e44b928c798ec (127.23.130.1:39669)
I20260808 15:47:15.395367 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fd690d030a4ee7b671e96a68af3527" candidate_uuid: "662f0d33c9814c9d8e552037e9e238dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e59e03157546c09ec3f7e3e32a8c14"
I20260808 15:47:15.395450 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fd690d030a4ee7b671e96a68af3527" candidate_uuid: "662f0d33c9814c9d8e552037e9e238dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9c2342924e4848a50e44b928c798ec"
I20260808 15:47:15.395605 13980 raft_consensus.cc:3037] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:15.395639 13835 raft_consensus.cc:3037] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:15.398897 13980 raft_consensus.cc:2454] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662f0d33c9814c9d8e552037e9e238dc in term 1.
I20260808 15:47:15.398886 13835 raft_consensus.cc:2454] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662f0d33c9814c9d8e552037e9e238dc in term 1.
I20260808 15:47:15.399348 14061 leader_election.cc:302] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e9c2342924e4848a50e44b928c798ec, 662f0d33c9814c9d8e552037e9e238dc, 96e59e03157546c09ec3f7e3e32a8c14; no voters:
I20260808 15:47:15.399648 14211 raft_consensus.cc:2781] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:15.400151 14211 raft_consensus.cc:686] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 1 LEADER]: Becoming Leader. State: Replica: 662f0d33c9814c9d8e552037e9e238dc, State: Running, Role: LEADER
I20260808 15:47:15.400631 14211 consensus_queue.cc:237] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } }
I20260808 15:47:15.405651 13669 catalog_manager.cc:5495] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 662f0d33c9814c9d8e552037e9e238dc (127.23.130.3). New cstate: current_term: 1 leader_uuid: "662f0d33c9814c9d8e552037e9e238dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662f0d33c9814c9d8e552037e9e238dc" member_type: VOTER last_known_addr { host: "127.23.130.3" port: 34559 } health_report { overall_health: HEALTHY } } }
W20260808 15:47:15.431334 13884 tablet.cc:2369] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:15.440007 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.434296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.23.130.3:34559)
0808 15:47:15.434516 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.434524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.434680 (+ 156us) server_negotiation.cc:407] Connection header received
0808 15:47:15.434930 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.434935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.435021 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.435182 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:15.435565 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.436308 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.437755 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.438196 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.438334 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.438889 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:15.439409 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:15.439576 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.439658 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.439739 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:47:15.448082 13980 raft_consensus.cc:1261] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 1 FOLLOWER]: Refusing update from remote peer 662f0d33c9814c9d8e552037e9e238dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:15.448083 13835 raft_consensus.cc:1261] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 1 FOLLOWER]: Refusing update from remote peer 662f0d33c9814c9d8e552037e9e238dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:15.449100 14214 consensus_queue.cc:1035] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e59e03157546c09ec3f7e3e32a8c14" member_type: VOTER last_known_addr { host: "127.23.130.2" port: 33169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:15.449469 14211 consensus_queue.cc:1035] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9c2342924e4848a50e44b928c798ec" member_type: VOTER last_known_addr { host: "127.23.130.1" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:15.461036 14217 mvcc.cc:205] Tried to move back new op lower bound from 7316291729185226752 to 7316291729004519424. Current Snapshot: MvccSnapshot[applied={T|T < 7316291729185226752}]
I20260808 15:47:15.475015 13669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:15.475277 13669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:47:15.478933 13669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f02702c3cf6487cbd183212b7ff02b7: Sending DeleteTablet for 3 replicas of tablet f3fd690d030a4ee7b671e96a68af3527
I20260808 15:47:15.480234 14111 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fd690d030a4ee7b671e96a68af3527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:15 UTC) from {username='slave'} at 127.0.0.1:33616
I20260808 15:47:15.480198 13956 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fd690d030a4ee7b671e96a68af3527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:15 UTC) from {username='slave'} at 127.0.0.1:42372
I20260808 15:47:15.480293 13815 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fd690d030a4ee7b671e96a68af3527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:47:15 UTC) from {username='slave'} at 127.0.0.1:50518
I20260808 15:47:15.484593 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13744
I20260808 15:47:15.495244 14228 tablet_replica.cc:330] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: stopping tablet replica
I20260808 15:47:15.495879 14228 raft_consensus.cc:2229] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:15.496582 14228 raft_consensus.cc:2258] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:15.499812 14228 ts_tablet_manager.cc:1905] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:15.503022 14229 tablet_replica.cc:330] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: stopping tablet replica
I20260808 15:47:15.503288 14227 tablet_replica.cc:330] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: stopping tablet replica
I20260808 15:47:15.503535 14229 raft_consensus.cc:2229] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:47:15.503724 14227 raft_consensus.cc:2229] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:47:15.504082 14229 raft_consensus.cc:2258] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:15.504480 14227 raft_consensus.cc:2258] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:47:15.506582 14227 ts_tablet_manager.cc:1905] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:15.508302 14229 ts_tablet_manager.cc:1905] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:47:15.513895 14228 ts_tablet_manager.cc:1918] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:15.514061 14227 ts_tablet_manager.cc:1918] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:15.514135 14228 log.cc:1196] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-0/wal/wals/f3fd690d030a4ee7b671e96a68af3527
I20260808 15:47:15.514232 14227 log.cc:1196] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-2/wal/wals/f3fd690d030a4ee7b671e96a68af3527
I20260808 15:47:15.514837 14227 ts_tablet_manager.cc:1939] T f3fd690d030a4ee7b671e96a68af3527 P 662f0d33c9814c9d8e552037e9e238dc: Deleting consensus metadata
I20260808 15:47:15.514988 14228 ts_tablet_manager.cc:1939] T f3fd690d030a4ee7b671e96a68af3527 P 4e9c2342924e4848a50e44b928c798ec: Deleting consensus metadata
I20260808 15:47:15.515290 14229 ts_tablet_manager.cc:1918] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:47:15.515450 14229 log.cc:1196] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Deleting WAL directory at /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786203706771851-24072-0/minicluster-data/ts-1/wal/wals/f3fd690d030a4ee7b671e96a68af3527
I20260808 15:47:15.516004 14229 ts_tablet_manager.cc:1939] T f3fd690d030a4ee7b671e96a68af3527 P 96e59e03157546c09ec3f7e3e32a8c14: Deleting consensus metadata
I20260808 15:47:15.516782 13655 catalog_manager.cc:4841] TS 662f0d33c9814c9d8e552037e9e238dc (127.23.130.3:34559): tablet f3fd690d030a4ee7b671e96a68af3527 (table test-table [id=d368a694c3714a51aa484043f37df369]) successfully deleted
I20260808 15:47:15.517215 13658 catalog_manager.cc:4841] TS 4e9c2342924e4848a50e44b928c798ec (127.23.130.1:39669): tablet f3fd690d030a4ee7b671e96a68af3527 (table test-table [id=d368a694c3714a51aa484043f37df369]) successfully deleted
I20260808 15:47:15.518509 13656 catalog_manager.cc:4841] TS 96e59e03157546c09ec3f7e3e32a8c14 (127.23.130.2:33169): tablet f3fd690d030a4ee7b671e96a68af3527 (table test-table [id=d368a694c3714a51aa484043f37df369]) successfully deleted
I20260808 15:47:15.522262 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.485751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.23.130.1:39669)
0808 15:47:15.486045 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.486054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.507410 (+ 21356us) server_negotiation.cc:407] Connection header received
0808 15:47:15.507664 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.507670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.507786 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.507890 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:15.516051 (+ 8161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.517227 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.519172 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.519562 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.519662 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.520902 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:15.520910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:15.520923 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:15.521495 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:15.521541 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:15.521559 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:15.521653 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.521819 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.521908 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260808 15:47:15.523000 13877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:15.615227 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13744
I20260808 15:47:15.637310 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13888
I20260808 15:47:15.646124 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.638180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.23.130.2:33169)
0808 15:47:15.638470 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.638478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.638503 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:47:15.638651 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.638657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.638780 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.638891 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:15.639270 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.640504 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.642316 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.643033 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.643167 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.644456 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:15.644466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:15.644482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:15.645247 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:15.645309 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:15.645334 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:15.645462 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.645617 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.645727 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:47:15.647130 14027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:15.759737 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13888
I20260808 15:47:15.777729 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14038
I20260808 15:47:15.785106 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:15.778627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.23.130.3:34559)
0808 15:47:15.778803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:15.778811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:15.778915 (+ 104us) server_negotiation.cc:407] Connection header received
0808 15:47:15.779141 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:15.779146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:15.779249 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:15.779370 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:15.779797 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.780613 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.781884 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:15.782327 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:15.782505 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:15.783657 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:15.783666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:15.783681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:15.784303 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:15.784348 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:15.784372 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:15.784504 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:15.784653 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 15:47:15.784755 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:47:15.785786 14177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:15.906538 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14038
I20260808 15:47:15.929425 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 13637
I20260808 15:47:15.930583 13712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:16.040736 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 13637
2026-08-08T15:47:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2981 ms)
[----------] 6 tests from EncryptionPolicyTest (23247 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:16 dist-test-slave-gp6n krb5kdc[14238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:16 dist-test-slave-gp6n krb5kdc[14238](info): set up 2 sockets
Aug 08 15:47:16 dist-test-slave-gp6n krb5kdc[14238](info): commencing operation
krb5kdc: starting...
W20260808 15:47:17.245898 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:17 dist-test-slave-gp6n krb5kdc[14238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:17.565456 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:17.908627 14252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:17.908890 14252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:17.909070 14252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:17.916213 14252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:17.916316 14252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:17.916363 14252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:17.916456 14252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:17.916522 14252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:17.926275 14252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:17.927575 14252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:17.928972 14252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:17.940886 14258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:17.941112 14257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:17.940929 14260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:17.941725 14252 server_base.cc:1034] running on GCE node
I20260808 15:47:17.942554 14252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:17.943997 14252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:17.945235 14252 hybrid_clock.cc:648] HybridClock initialized: now 1786204037945194 us; error 45 us; skew 500 ppm
I20260808 15:47:17.945586 14252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:17 dist-test-slave-gp6n krb5kdc[14238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:17.948803 14252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:17.950191 14252 webserver.cc:460] Webserver started at http://127.0.0.1:38665/ using document root <none> and password file <none>
I20260808 15:47:17.950917 14252 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:17.951068 14252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:17.951377 14252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:17.954504 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "d85a458d4a254d048b69bb834dbd312d"
format_stamp: "Formatted at 2026-08-08 15:47:17 on dist-test-slave-gp6n"
server_key: "58b223e97496ea0a7280831a68513a8d"
server_key_iv: "ffc72947e7c0104ae64f265858ee5987"
server_key_version: "encryptionkey@0"
I20260808 15:47:17.955314 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "d85a458d4a254d048b69bb834dbd312d"
format_stamp: "Formatted at 2026-08-08 15:47:17 on dist-test-slave-gp6n"
server_key: "58b223e97496ea0a7280831a68513a8d"
server_key_iv: "ffc72947e7c0104ae64f265858ee5987"
server_key_version: "encryptionkey@0"
I20260808 15:47:17.960539 14252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:47:17.964077 14267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:17.965389 14252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:47:17.965583 14252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "d85a458d4a254d048b69bb834dbd312d"
format_stamp: "Formatted at 2026-08-08 15:47:17 on dist-test-slave-gp6n"
server_key: "58b223e97496ea0a7280831a68513a8d"
server_key_iv: "ffc72947e7c0104ae64f265858ee5987"
server_key_version: "encryptionkey@0"
I20260808 15:47:17.965785 14252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:18.010210 14252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:18.015022 14252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:18.015322 14252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:18.034616 14252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37443
I20260808 15:47:18.034651 14322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37443 every 8 connection(s)
I20260808 15:47:18.036015 14252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:18.039990 14324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:18.045707 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14252
I20260808 15:47:18.045950 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:18.046856 24072 external_mini_cluster.cc:1428] Setting key 729809c35ebcc02058aaa930427b10a7
I20260808 15:47:18.052695 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d: Bootstrap starting.
I20260808 15:47:18.056672 14324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:18.057940 14324 log.cc:824] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:18.061493 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d: No bootstrap required, opened a new log
Aug 08 15:47:18 dist-test-slave-gp6n krb5kdc[14238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:47:18.072574 14324 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260808 15:47:18.072921 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:18.048940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59249 (local address 127.0.0.1:37443)
0808 15:47:18.049830 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:18.049842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:18.049887 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:47:18.051716 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:18.051738 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:18.052129 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:18.052450 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:18.054382 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:18.056017 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:18.057900 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:18.058886 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:18.059106 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:18.064013 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:18.064023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:18.064042 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:18.064086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:18.068181 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:18.069023 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:18.069031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:18.069040 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:18.069268 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:18.069712 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:18.069717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:18.069720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:18.070549 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:18.070901 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:18.071240 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:47:18.071587 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":449,"threads_started":1}
I20260808 15:47:18.073035 14324 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:18.073134 14324 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85a458d4a254d048b69bb834dbd312d, State: Initialized, Role: FOLLOWER
I20260808 15:47:18.073815 14324 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260808 15:47:18.074110 14324 raft_consensus.cc:388] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:18.074256 14324 raft_consensus.cc:482] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:18.074530 14324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:18.076963 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:18.077311 14324 raft_consensus.cc:504] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260808 15:47:18.077801 14324 leader_election.cc:302] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85a458d4a254d048b69bb834dbd312d; no voters:
I20260808 15:47:18.078503 14324 leader_election.cc:288] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:18.078630 14332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:18.085654 14332 raft_consensus.cc:686] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [term 1 LEADER]: Becoming Leader. State: Replica: d85a458d4a254d048b69bb834dbd312d, State: Running, Role: LEADER
I20260808 15:47:18.086272 14332 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260808 15:47:18.086995 14324 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:18.096822 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:18.081919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55751 (local address 127.0.0.1:37443)
0808 15:47:18.082142 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:18.082151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:18.082734 (+ 583us) server_negotiation.cc:407] Connection header received
0808 15:47:18.082938 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:18.082944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:18.083019 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:18.083121 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:18.084540 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:18.085658 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:18.087103 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:18.087559 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:18.087713 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:18.090445 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:18.090452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:18.090463 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:18.090526 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:18.093767 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:18.094603 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:18.094610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:18.094612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:18.094729 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:18.095191 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:18.095196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:18.095199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:18.095531 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:18.095807 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:18.096220 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 15:47:18.096387 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:47:18.098078 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85a458d4a254d048b69bb834dbd312d. Latest consensus state: current_term: 1 leader_uuid: "d85a458d4a254d048b69bb834dbd312d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260808 15:47:18.098603 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:18.101791 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d85a458d4a254d048b69bb834dbd312d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a458d4a254d048b69bb834dbd312d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260808 15:47:18.102108 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:18.102298 14349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:18.108013 14349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:18.116690 14349 catalog_manager.cc:1347] Generated new cluster ID: 3fad761745f24a80bc6f001193253dd1
I20260808 15:47:18.116801 14349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:18.134155 14349 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:18.136118 14349 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:18.152460 14349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d85a458d4a254d048b69bb834dbd312d: Generated new TSK 0
I20260808 15:47:18.153293 14349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:18.167776 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14252
I20260808 15:47:18.169157 14318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:18.268966 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14252
W20260808 15:47:18.290848 14236 connection.cc:535] client connection to 127.0.0.1:37443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37443): Connection reset by peer (error 104)
2026-08-08T15:47:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:18 dist-test-slave-gp6n krb5kdc[14362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:18 dist-test-slave-gp6n krb5kdc[14362](info): set up 2 sockets
Aug 08 15:47:18 dist-test-slave-gp6n krb5kdc[14362](info): commencing operation
krb5kdc: starting...
W20260808 15:47:19.456326 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.004s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:19 dist-test-slave-gp6n krb5kdc[14362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:19.777300 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:20.125732 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:20.126037 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:20.126164 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:20.133589 14375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:20.133692 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:20.133795 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:20.133898 14375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:20.133983 14375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:20.143790 14375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:20.145102 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:20.146456 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:20.158694 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:20.158813 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:20.158833 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:20.159999 14375 server_base.cc:1034] running on GCE node
I20260808 15:47:20.160756 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:20.162186 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:20.163415 14375 hybrid_clock.cc:648] HybridClock initialized: now 1786204040163382 us; error 56 us; skew 500 ppm
I20260808 15:47:20.163811 14375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:20 dist-test-slave-gp6n krb5kdc[14362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:20.172895 14375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:20.174633 14375 webserver.cc:460] Webserver started at http://127.0.0.1:42981/ using document root <none> and password file <none>
I20260808 15:47:20.175333 14375 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:20.175485 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:20.175801 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:20.178900 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "7457f18944c64377a3087e729e5e30b3"
format_stamp: "Formatted at 2026-08-08 15:47:20 on dist-test-slave-gp6n"
server_key: "c121fc145dc7c69f0ca14a1d96dd090b"
server_key_iv: "d402a6d960e48c76244a761fe75a33fa"
server_key_version: "encryptionkey@0"
I20260808 15:47:20.179683 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "7457f18944c64377a3087e729e5e30b3"
format_stamp: "Formatted at 2026-08-08 15:47:20 on dist-test-slave-gp6n"
server_key: "c121fc145dc7c69f0ca14a1d96dd090b"
server_key_iv: "d402a6d960e48c76244a761fe75a33fa"
server_key_version: "encryptionkey@0"
I20260808 15:47:20.184870 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:47:20.188442 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:20.189785 14375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:47:20.189967 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "7457f18944c64377a3087e729e5e30b3"
format_stamp: "Formatted at 2026-08-08 15:47:20 on dist-test-slave-gp6n"
server_key: "c121fc145dc7c69f0ca14a1d96dd090b"
server_key_iv: "d402a6d960e48c76244a761fe75a33fa"
server_key_version: "encryptionkey@0"
I20260808 15:47:20.190161 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:20.210690 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:20.214758 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:20.215065 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:20.239890 14375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38145
I20260808 15:47:20.239921 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38145 every 8 connection(s)
I20260808 15:47:20.241493 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:20.245455 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:20.249755 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14375
I20260808 15:47:20.249960 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:20.250563 24072 external_mini_cluster.cc:1428] Setting key eb0bd63e77edecb5268b6037bcf72321
I20260808 15:47:20.259291 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3: Bootstrap starting.
I20260808 15:47:20.263316 14447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:20.264482 14447 log.cc:824] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:20.264719 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:20.252514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46213 (local address 127.0.0.1:38145)
0808 15:47:20.253353 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:20.253366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:20.253406 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:20.255135 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:20.255161 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:20.255619 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:20.255978 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:20.256627 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:20.257973 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:20.259940 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:20.260615 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:20.260754 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:20.262177 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:20.262185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:20.262199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:20.262827 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:20.262903 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:20.262929 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:20.263063 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:20.263377 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 15:47:20.263588 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":417,"threads_started":1}
I20260808 15:47:20.267617 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3: No bootstrap required, opened a new log
I20260808 15:47:20.268707 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:20.278187 14447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260808 15:47:20.278841 14447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:20.279090 14447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7457f18944c64377a3087e729e5e30b3, State: Initialized, Role: FOLLOWER
I20260808 15:47:20.279817 14447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260808 15:47:20.280143 14447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:20.280318 14447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:20.280526 14447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:20.281500 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:20.274294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47513 (local address 127.0.0.1:38145)
0808 15:47:20.274502 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:20.274509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:20.274893 (+ 384us) server_negotiation.cc:407] Connection header received
0808 15:47:20.275058 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:20.275063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:20.275145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:20.275273 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:20.275787 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:20.276973 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:20.278278 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:20.279121 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:20.279281 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:20.280682 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:20.280690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:20.280698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:20.280726 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:20.280795 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:20.280820 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:20.280960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:20.281093 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 15:47:20.281149 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:47:20.283368 14447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260808 15:47:20.283856 14447 leader_election.cc:302] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7457f18944c64377a3087e729e5e30b3; no voters:
I20260808 15:47:20.284379 14447 leader_election.cc:288] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:20.284520 14459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:20.285393 14459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [term 1 LEADER]: Becoming Leader. State: Replica: 7457f18944c64377a3087e729e5e30b3, State: Running, Role: LEADER
I20260808 15:47:20.286046 14459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260808 15:47:20.286826 14447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:20.289554 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7457f18944c64377a3087e729e5e30b3. Latest consensus state: current_term: 1 leader_uuid: "7457f18944c64377a3087e729e5e30b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } } }
I20260808 15:47:20.290057 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:20.290257 14460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7457f18944c64377a3087e729e5e30b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7457f18944c64377a3087e729e5e30b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } } }
I20260808 15:47:20.290879 14460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:20.292737 14467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:20.299594 14467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:20.313051 14467 catalog_manager.cc:1347] Generated new cluster ID: 022a5ecffbb84e09858aa6083283e946
I20260808 15:47:20.313242 14467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:20.345115 14467 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:20.346581 14467 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:20.354031 14467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7457f18944c64377a3087e729e5e30b3: Generated new TSK 0
I20260808 15:47:20.355106 14467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:20.373111 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14375
W20260808 15:47:20.374375 14441 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46213
W20260808 15:47:20.375429 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:20.375654 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14375
W20260808 15:47:20.397048 14359 connection.cc:535] client connection to 127.0.0.1:38145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38145): Connection reset by peer (error 104)
2026-08-08T15:47:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:20 dist-test-slave-gp6n krb5kdc[14482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:20 dist-test-slave-gp6n krb5kdc[14482](info): set up 2 sockets
Aug 08 15:47:20 dist-test-slave-gp6n krb5kdc[14482](info): commencing operation
krb5kdc: starting...
W20260808 15:47:21.613680 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:21 dist-test-slave-gp6n krb5kdc[14482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:21.929774 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:22.279271 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:22.279572 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:22.279742 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:22.286859 14499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:22.286964 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:22.287086 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:22.287190 14499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:22.287269 14499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:22.296957 14499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:22.298286 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:22.299650 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:22.311571 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:22.311655 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:22.311601 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:22.312594 14499 server_base.cc:1034] running on GCE node
I20260808 15:47:22.313551 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:22.315285 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:22.316573 14499 hybrid_clock.cc:648] HybridClock initialized: now 1786204042316515 us; error 57 us; skew 500 ppm
I20260808 15:47:22.316885 14499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:22 dist-test-slave-gp6n krb5kdc[14482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:22.320048 14499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:22.321552 14499 webserver.cc:460] Webserver started at http://127.0.0.1:39771/ using document root <none> and password file <none>
I20260808 15:47:22.322223 14499 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:22.322371 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:22.322702 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:22.325698 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "8907d7aa68d1464cb5800f8752a3ee67"
format_stamp: "Formatted at 2026-08-08 15:47:22 on dist-test-slave-gp6n"
server_key: "170b9c5e2f74fa8817124b6615d388fd"
server_key_iv: "456c9e8164baa5b9708c92a2249716a6"
server_key_version: "encryptionkey@0"
I20260808 15:47:22.326534 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "8907d7aa68d1464cb5800f8752a3ee67"
format_stamp: "Formatted at 2026-08-08 15:47:22 on dist-test-slave-gp6n"
server_key: "170b9c5e2f74fa8817124b6615d388fd"
server_key_iv: "456c9e8164baa5b9708c92a2249716a6"
server_key_version: "encryptionkey@0"
I20260808 15:47:22.331549 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:47:22.335000 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:22.336432 14499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:47:22.336608 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "8907d7aa68d1464cb5800f8752a3ee67"
format_stamp: "Formatted at 2026-08-08 15:47:22 on dist-test-slave-gp6n"
server_key: "170b9c5e2f74fa8817124b6615d388fd"
server_key_iv: "456c9e8164baa5b9708c92a2249716a6"
server_key_version: "encryptionkey@0"
I20260808 15:47:22.336776 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:22.342628 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:22.347036 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:22.347352 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:22.366812 14499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38417
I20260808 15:47:22.366870 14569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38417 every 8 connection(s)
I20260808 15:47:22.368283 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:22.370026 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14499
I20260808 15:47:22.370217 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:22.370770 24072 external_mini_cluster.cc:1428] Setting key 3d21b674055ed0a23d38614c3ff9a2d7
I20260808 15:47:22.372493 14570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:22.378770 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:22.372603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60811 (local address 127.0.0.1:38417)
0808 15:47:22.373391 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:22.373403 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:22.373443 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:22.375191 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:22.375213 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:22.375564 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:22.375982 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:22.377090 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:22.377096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:22.377111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:22.377127 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:22.377195 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:22.377218 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:22.377475 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:22.377622 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 15:47:22.377813 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":356,"threads_started":1}
I20260808 15:47:22.382450 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:22.384712 14570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67: Bootstrap starting.
I20260808 15:47:22.388213 14570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:22.389253 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:22.386860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37147 (local address 127.0.0.1:38417)
0808 15:47:22.387034 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:22.387039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:22.387364 (+ 325us) server_negotiation.cc:407] Connection header received
0808 15:47:22.387491 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:22.387495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:22.387554 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:22.387657 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:22.388597 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:22.388603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:22.388609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:22.388636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:22.388692 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:22.388716 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:22.388831 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:22.388905 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 15:47:22.388965 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:47:22.389581 14570 log.cc:824] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:22.392409 14570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67: No bootstrap required, opened a new log
I20260808 15:47:22.399940 14570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } }
I20260808 15:47:22.400312 14570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:22.400413 14570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8907d7aa68d1464cb5800f8752a3ee67, State: Initialized, Role: FOLLOWER
I20260808 15:47:22.400954 14570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } }
I20260808 15:47:22.401173 14570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:22.401324 14570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:22.401496 14570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:22.403836 14570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } }
I20260808 15:47:22.404338 14570 leader_election.cc:302] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8907d7aa68d1464cb5800f8752a3ee67; no voters:
I20260808 15:47:22.404948 14570 leader_election.cc:288] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:22.405090 14580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:22.405851 14580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [term 1 LEADER]: Becoming Leader. State: Replica: 8907d7aa68d1464cb5800f8752a3ee67, State: Running, Role: LEADER
I20260808 15:47:22.406364 14580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } }
I20260808 15:47:22.407123 14570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:22.409475 14582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8907d7aa68d1464cb5800f8752a3ee67. Latest consensus state: current_term: 1 leader_uuid: "8907d7aa68d1464cb5800f8752a3ee67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } } }
I20260808 15:47:22.409865 14582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:22.410609 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8907d7aa68d1464cb5800f8752a3ee67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907d7aa68d1464cb5800f8752a3ee67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38417 } } }
I20260808 15:47:22.410887 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:22.415234 14586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:22.421123 14586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:22.430105 14586 catalog_manager.cc:1347] Generated new cluster ID: 0122dd112605433aa8afc285693a24e8
I20260808 15:47:22.430231 14586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:22.440830 14586 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:22.441247 14586 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:22.447322 14586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8907d7aa68d1464cb5800f8752a3ee67: Generated new TSK 0
I20260808 15:47:22.448308 14586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:22.458639 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14499
W20260808 15:47:22.459995 14565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60811
W20260808 15:47:22.460718 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:22.460875 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14499
2026-08-08T15:47:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:22 dist-test-slave-gp6n krb5kdc[14602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:22 dist-test-slave-gp6n krb5kdc[14602](info): set up 2 sockets
Aug 08 15:47:22 dist-test-slave-gp6n krb5kdc[14602](info): commencing operation
krb5kdc: starting...
W20260808 15:47:23.665606 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:23 dist-test-slave-gp6n krb5kdc[14602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:23.981264 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:24.333586 14617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:24.333952 14617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:24.334115 14617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:24.341691 14617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:24.341837 14617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:24.341948 14617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:24.342060 14617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:24.342146 14617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:24.352305 14617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:24.353686 14617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:24.355077 14617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:24.367265 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:24.367276 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:24.367509 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:24.368032 14617 server_base.cc:1034] running on GCE node
I20260808 15:47:24.368906 14617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:24.370340 14617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:24.371603 14617 hybrid_clock.cc:648] HybridClock initialized: now 1786204044371544 us; error 58 us; skew 500 ppm
I20260808 15:47:24.371964 14617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:24 dist-test-slave-gp6n krb5kdc[14602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:24.375320 14617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:24.376718 14617 webserver.cc:460] Webserver started at http://127.0.0.1:39099/ using document root <none> and password file <none>
I20260808 15:47:24.377347 14617 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:24.377453 14617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:24.377713 14617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:24.380666 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "580a019cb2be482fbdf8b0b16fd426de"
format_stamp: "Formatted at 2026-08-08 15:47:24 on dist-test-slave-gp6n"
server_key: "29874779d3307c05af6408f1810305de"
server_key_iv: "5f113ca25be134dbf339d88b3d430d3c"
server_key_version: "encryptionkey@0"
I20260808 15:47:24.381480 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "580a019cb2be482fbdf8b0b16fd426de"
format_stamp: "Formatted at 2026-08-08 15:47:24 on dist-test-slave-gp6n"
server_key: "29874779d3307c05af6408f1810305de"
server_key_iv: "5f113ca25be134dbf339d88b3d430d3c"
server_key_version: "encryptionkey@0"
I20260808 15:47:24.386632 14617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:47:24.389957 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:24.391419 14617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:24.391618 14617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "580a019cb2be482fbdf8b0b16fd426de"
format_stamp: "Formatted at 2026-08-08 15:47:24 on dist-test-slave-gp6n"
server_key: "29874779d3307c05af6408f1810305de"
server_key_iv: "5f113ca25be134dbf339d88b3d430d3c"
server_key_version: "encryptionkey@0"
I20260808 15:47:24.391822 14617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:24.405917 14617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:24.410599 14617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:24.410903 14617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:24.429844 14617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46473
I20260808 15:47:24.429875 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46473 every 8 connection(s)
I20260808 15:47:24.431283 14617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:24.432199 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14617
I20260808 15:47:24.432402 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:24.432963 24072 external_mini_cluster.cc:1428] Setting key 03ad6d53f91a562f854e22dbab292ff4
I20260808 15:47:24.435956 14693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:47:24 dist-test-slave-gp6n krb5kdc[14602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:47:24.446682 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de: Bootstrap starting.
I20260808 15:47:24.450922 14693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:24.452243 14693 log.cc:824] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:24.456075 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de: No bootstrap required, opened a new log
I20260808 15:47:24.456872 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:24.434916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50831 (local address 127.0.0.1:46473)
0808 15:47:24.435672 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:24.435684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:24.435722 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:47:24.437393 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:24.437414 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:24.437771 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:24.438159 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:24.439643 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:24.441150 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:24.442440 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:24.443114 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:24.443303 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:24.447536 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:24.447546 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:24.447564 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:24.447608 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:24.451714 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:24.452566 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:24.452576 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:24.452584 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:24.452795 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:24.453378 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:24.453383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:24.453385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:24.454157 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:24.454510 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:24.454958 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 15:47:24.455425 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":337,"threads_started":1}
I20260808 15:47:24.460904 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:24.466394 14693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260808 15:47:24.466990 14693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:24.467123 14693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580a019cb2be482fbdf8b0b16fd426de, State: Initialized, Role: FOLLOWER
I20260808 15:47:24.467924 14693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260808 15:47:24.468273 14693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:24.468446 14693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:24.468664 14693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:24.472092 14693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260808 15:47:24.472653 14693 leader_election.cc:302] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580a019cb2be482fbdf8b0b16fd426de; no voters:
I20260808 15:47:24.473400 14693 leader_election.cc:288] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:24.473523 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:24.474843 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [term 1 LEADER]: Becoming Leader. State: Replica: 580a019cb2be482fbdf8b0b16fd426de, State: Running, Role: LEADER
I20260808 15:47:24.475561 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260808 15:47:24.476223 14693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:24.479077 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 580a019cb2be482fbdf8b0b16fd426de. Latest consensus state: current_term: 1 leader_uuid: "580a019cb2be482fbdf8b0b16fd426de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260808 15:47:24.479044 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "580a019cb2be482fbdf8b0b16fd426de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580a019cb2be482fbdf8b0b16fd426de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260808 15:47:24.479565 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:24.479599 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:24.483414 14709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:24.486454 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:24.465461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49113 (local address 127.0.0.1:46473)
0808 15:47:24.465670 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:24.465677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:24.466292 (+ 615us) server_negotiation.cc:407] Connection header received
0808 15:47:24.466535 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:24.466539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:24.466606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:24.466697 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:24.468922 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:24.469737 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:24.471772 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:24.472224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:24.472366 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:24.475460 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:24.475465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:24.475470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:24.475503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:24.479329 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:24.480938 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:24.480948 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:24.480955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:24.482956 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:24.483657 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:24.483667 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:24.483673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:24.484230 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:24.484608 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:24.485496 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 15:47:24.485851 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:47:24.494876 14709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:24.508962 14709 catalog_manager.cc:1347] Generated new cluster ID: 684a8fc5d8cf47be882cce4df2fc74b4
I20260808 15:47:24.509092 14709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:24.533479 14709 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:24.534988 14709 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:24.543996 14709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 580a019cb2be482fbdf8b0b16fd426de: Generated new TSK 0
I20260808 15:47:24.544854 14709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:24.553452 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14617
I20260808 15:47:24.554579 14688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:24.650002 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14617
2026-08-08T15:47:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:24 dist-test-slave-gp6n krb5kdc[14725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:24 dist-test-slave-gp6n krb5kdc[14725](info): set up 2 sockets
Aug 08 15:47:24 dist-test-slave-gp6n krb5kdc[14725](info): commencing operation
krb5kdc: starting...
W20260808 15:47:25.849872 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:26 dist-test-slave-gp6n krb5kdc[14725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:26.177613 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:26.528658 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:26.528947 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:26.529043 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:26.536149 14741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:26.536259 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:26.536306 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:26.536368 14741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:26.536412 14741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:26.546141 14741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:26.547449 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:26.548799 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:26.560788 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:26.560849 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:26.560853 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:26.562067 14741 server_base.cc:1034] running on GCE node
I20260808 15:47:26.562983 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:26.564422 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:26.565625 14741 hybrid_clock.cc:648] HybridClock initialized: now 1786204046565578 us; error 39 us; skew 500 ppm
I20260808 15:47:26.565979 14741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:26 dist-test-slave-gp6n krb5kdc[14725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:26.573688 14741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:26.575361 14741 webserver.cc:460] Webserver started at http://127.0.0.1:37341/ using document root <none> and password file <none>
I20260808 15:47:26.576005 14741 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:26.576110 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:26.576352 14741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:26.579368 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "20089bf429e448b9b8a4a6cdcd9aec89"
format_stamp: "Formatted at 2026-08-08 15:47:26 on dist-test-slave-gp6n"
server_key: "b1d1238a9631c7ed0d238844e4ba7102"
server_key_iv: "6c00b84210996ad84a51c0662a899f30"
server_key_version: "encryptionkey@0"
I20260808 15:47:26.580103 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "20089bf429e448b9b8a4a6cdcd9aec89"
format_stamp: "Formatted at 2026-08-08 15:47:26 on dist-test-slave-gp6n"
server_key: "b1d1238a9631c7ed0d238844e4ba7102"
server_key_iv: "6c00b84210996ad84a51c0662a899f30"
server_key_version: "encryptionkey@0"
I20260808 15:47:26.585233 14741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:47:26.588640 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:26.590073 14741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:47:26.590267 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "20089bf429e448b9b8a4a6cdcd9aec89"
format_stamp: "Formatted at 2026-08-08 15:47:26 on dist-test-slave-gp6n"
server_key: "b1d1238a9631c7ed0d238844e4ba7102"
server_key_iv: "6c00b84210996ad84a51c0662a899f30"
server_key_version: "encryptionkey@0"
I20260808 15:47:26.590529 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:26.609587 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:26.613538 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:26.613775 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:26.633374 14741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45679
I20260808 15:47:26.633482 14811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45679 every 8 connection(s)
I20260808 15:47:26.634774 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:26.638144 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14741
I20260808 15:47:26.638337 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:26.638823 24072 external_mini_cluster.cc:1428] Setting key 9bfb09a0bc1bedc72709a26ece905b28
I20260808 15:47:26.638821 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:26.657931 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:26.640671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53235 (local address 127.0.0.1:45679)
0808 15:47:26.642205 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:26.642221 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:26.642277 (+ 56us) server_negotiation.cc:407] Connection header received
0808 15:47:26.645438 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:26.645466 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:26.646012 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:26.646480 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:26.647036 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:26.649225 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:26.651421 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:26.652581 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:26.652840 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:26.654166 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:26.654179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:26.654202 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:26.655203 (+ 1001us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:26.655333 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:26.655383 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:26.655540 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:26.655895 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 15:47:26.656215 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1230,"thread_start_us":590,"threads_started":1}
I20260808 15:47:26.660600 14812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: Bootstrap starting.
I20260808 15:47:26.662061 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:26.664111 14812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:26.665444 14812 log.cc:824] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:26.669507 14812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: No bootstrap required, opened a new log
I20260808 15:47:26.676393 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:26.667178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43707 (local address 127.0.0.1:45679)
0808 15:47:26.667455 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:26.667462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:26.668243 (+ 781us) server_negotiation.cc:407] Connection header received
0808 15:47:26.668361 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:26.668369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:26.668471 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:26.668610 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:26.669661 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:26.670610 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:26.672888 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:26.673619 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:26.673768 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:26.675479 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:26.675485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:26.675490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:26.675514 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:26.675570 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:26.675591 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:26.675682 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:26.675892 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 15:47:26.675998 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:47:26.680684 14812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } }
I20260808 15:47:26.681202 14812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:26.681309 14812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20089bf429e448b9b8a4a6cdcd9aec89, State: Initialized, Role: FOLLOWER
I20260808 15:47:26.681960 14812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } }
I20260808 15:47:26.682207 14812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:26.682379 14812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:26.682570 14812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:26.684798 14812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } }
I20260808 15:47:26.685305 14812 leader_election.cc:302] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20089bf429e448b9b8a4a6cdcd9aec89; no voters:
I20260808 15:47:26.685990 14812 leader_election.cc:288] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:26.686183 14822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:26.688098 14812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:26.692943 14822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [term 1 LEADER]: Becoming Leader. State: Replica: 20089bf429e448b9b8a4a6cdcd9aec89, State: Running, Role: LEADER
I20260808 15:47:26.693660 14822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } }
I20260808 15:47:26.696956 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } } }
I20260808 15:47:26.697424 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:26.697273 14824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20089bf429e448b9b8a4a6cdcd9aec89. Latest consensus state: current_term: 1 leader_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20089bf429e448b9b8a4a6cdcd9aec89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45679 } } }
I20260808 15:47:26.697595 14824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89 [sys.catalog]: This master's current role is: LEADER
W20260808 15:47:26.704643 14839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:47:26.704771 14839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:47:26.704916 14841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:26.707450 14841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:26.717305 14841 catalog_manager.cc:1347] Generated new cluster ID: 0686e4b84ce44a0483f894659999c9ff
I20260808 15:47:26.717475 14841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:26.736191 14841 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:26.738111 14841 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:26.754207 14841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20089bf429e448b9b8a4a6cdcd9aec89: Generated new TSK 0
I20260808 15:47:26.755298 14841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:26.771521 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14741
W20260808 15:47:26.772768 14807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53235
W20260808 15:47:26.773715 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:26.773898 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14741
2026-08-08T15:47:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:26 dist-test-slave-gp6n krb5kdc[14847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:26 dist-test-slave-gp6n krb5kdc[14847](info): set up 2 sockets
Aug 08 15:47:26 dist-test-slave-gp6n krb5kdc[14847](info): commencing operation
krb5kdc: starting...
W20260808 15:47:27.963711 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:28 dist-test-slave-gp6n krb5kdc[14847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:28.293155 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:28.638130 14862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:28.638506 14862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:28.638623 14862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:28.645885 14862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:28.645999 14862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:28.646049 14862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:28.646119 14862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:28.646163 14862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:28.656361 14862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:28.657678 14862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:28.658974 14862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:28.670890 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:28.670935 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:28.670964 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:28.671663 14862 server_base.cc:1034] running on GCE node
I20260808 15:47:28.672472 14862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:28.673820 14862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:28.675057 14862 hybrid_clock.cc:648] HybridClock initialized: now 1786204048675014 us; error 55 us; skew 500 ppm
I20260808 15:47:28.675418 14862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:28 dist-test-slave-gp6n krb5kdc[14847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:28.678736 14862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:28.680178 14862 webserver.cc:460] Webserver started at http://127.0.0.1:43121/ using document root <none> and password file <none>
I20260808 15:47:28.680796 14862 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:28.680896 14862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:28.681140 14862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:28.684059 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "4d8c894c69844360bc30ee878850633f"
format_stamp: "Formatted at 2026-08-08 15:47:28 on dist-test-slave-gp6n"
server_key: "8e23398168d20408e239ff185079a3fa"
server_key_iv: "334942ef9bc455fad312ddc30a1c1019"
server_key_version: "encryptionkey@0"
I20260808 15:47:28.684806 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "4d8c894c69844360bc30ee878850633f"
format_stamp: "Formatted at 2026-08-08 15:47:28 on dist-test-slave-gp6n"
server_key: "8e23398168d20408e239ff185079a3fa"
server_key_iv: "334942ef9bc455fad312ddc30a1c1019"
server_key_version: "encryptionkey@0"
I20260808 15:47:28.689723 14862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:47:28.693082 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:28.694339 14862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:47:28.694559 14862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "4d8c894c69844360bc30ee878850633f"
format_stamp: "Formatted at 2026-08-08 15:47:28 on dist-test-slave-gp6n"
server_key: "8e23398168d20408e239ff185079a3fa"
server_key_iv: "334942ef9bc455fad312ddc30a1c1019"
server_key_version: "encryptionkey@0"
I20260808 15:47:28.694733 14862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:28.700438 14862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:28.704644 14862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:28.704888 14862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:28.723857 14862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35175
I20260808 15:47:28.723951 14939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35175 every 8 connection(s)
I20260808 15:47:28.725291 14862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:28.729115 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:28.734505 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14862
I20260808 15:47:28.734688 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:28.735213 24072 external_mini_cluster.cc:1428] Setting key a40913ab42f82e22c813d5327a5389d0
I20260808 15:47:28.740398 14940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f: Bootstrap starting.
I20260808 15:47:28.742908 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:28.736993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33243 (local address 127.0.0.1:35175)
0808 15:47:28.737783 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:28.737794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:28.737841 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:47:28.739566 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:28.739589 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:28.739924 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:28.740251 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:28.741307 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:28.741313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:28.741325 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:28.741338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:28.741398 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:28.741417 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:28.741582 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:28.741714 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 15:47:28.741902 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":352,"threads_started":1}
I20260808 15:47:28.743932 14940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:28.745191 14940 log.cc:824] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:28.746719 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:28.748092 14940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f: No bootstrap required, opened a new log
I20260808 15:47:28.753829 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:28.751253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44427 (local address 127.0.0.1:35175)
0808 15:47:28.751445 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:28.751451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:28.751724 (+ 273us) server_negotiation.cc:407] Connection header received
0808 15:47:28.751904 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:28.751908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:28.751953 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:28.752037 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:28.753052 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:28.753060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:28.753071 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:28.753093 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:28.753152 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:28.753175 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:28.753311 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:28.753451 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 15:47:28.753514 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:47:28.757368 14940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260808 15:47:28.757747 14940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:28.757871 14940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c894c69844360bc30ee878850633f, State: Initialized, Role: FOLLOWER
I20260808 15:47:28.758605 14940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260808 15:47:28.758860 14940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:28.759066 14940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:28.759266 14940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:28.761960 14940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260808 15:47:28.762558 14940 leader_election.cc:302] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d8c894c69844360bc30ee878850633f; no voters:
I20260808 15:47:28.763137 14940 leader_election.cc:288] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:28.763274 14950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:28.764068 14950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c894c69844360bc30ee878850633f, State: Running, Role: LEADER
I20260808 15:47:28.764662 14950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260808 15:47:28.765352 14940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:28.767768 14951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d8c894c69844360bc30ee878850633f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260808 15:47:28.767729 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8c894c69844360bc30ee878850633f. Latest consensus state: current_term: 1 leader_uuid: "4d8c894c69844360bc30ee878850633f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8c894c69844360bc30ee878850633f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260808 15:47:28.768318 14951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:28.768325 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:28.770874 14958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:28.776615 14958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:28.785354 14958 catalog_manager.cc:1347] Generated new cluster ID: 67e5932549a04f2cb43aeea35ae94d0d
I20260808 15:47:28.785470 14958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:28.806905 14958 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:28.807344 14958 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:28.820348 14958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d8c894c69844360bc30ee878850633f: Generated new TSK 0
I20260808 15:47:28.821187 14958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:28.831512 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14862
W20260808 15:47:28.832729 14934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33243
W20260808 15:47:28.833487 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:28.833645 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14862
2026-08-08T15:47:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:28 dist-test-slave-gp6n krb5kdc[14972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:28 dist-test-slave-gp6n krb5kdc[14972](info): set up 2 sockets
Aug 08 15:47:28 dist-test-slave-gp6n krb5kdc[14972](info): commencing operation
krb5kdc: starting...
W20260808 15:47:30.032989 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:30 dist-test-slave-gp6n krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:30.366395 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:30.720731 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:30.721024 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:30.721230 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:30.728498 14986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:30.728596 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:30.728706 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:30.728827 14986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:30.728904 14986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:30.738709 14986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:30.739948 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:30.741225 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:30.753163 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:30.753132 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:30.753329 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:30.754854 14986 server_base.cc:1034] running on GCE node
I20260808 15:47:30.755671 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:30.757107 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:30.758337 14986 hybrid_clock.cc:648] HybridClock initialized: now 1786204050758292 us; error 58 us; skew 500 ppm
I20260808 15:47:30.758775 14986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:30 dist-test-slave-gp6n krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:30.762241 14986 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:30.763777 14986 webserver.cc:460] Webserver started at http://127.0.0.1:38889/ using document root <none> and password file <none>
I20260808 15:47:30.764479 14986 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:30.764629 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:30.764931 14986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:30.768021 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "498180e1b6174336a04e7e78064f0bcc"
format_stamp: "Formatted at 2026-08-08 15:47:30 on dist-test-slave-gp6n"
server_key: "344ac824e2ed437ba3e7593551a8e09a"
server_key_iv: "3cf5efc43834fe06a63ae5f2957fc079"
server_key_version: "encryptionkey@0"
I20260808 15:47:30.768810 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "498180e1b6174336a04e7e78064f0bcc"
format_stamp: "Formatted at 2026-08-08 15:47:30 on dist-test-slave-gp6n"
server_key: "344ac824e2ed437ba3e7593551a8e09a"
server_key_iv: "3cf5efc43834fe06a63ae5f2957fc079"
server_key_version: "encryptionkey@0"
I20260808 15:47:30.773865 14986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:47:30.777275 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:30.778605 14986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:47:30.778780 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "498180e1b6174336a04e7e78064f0bcc"
format_stamp: "Formatted at 2026-08-08 15:47:30 on dist-test-slave-gp6n"
server_key: "344ac824e2ed437ba3e7593551a8e09a"
server_key_iv: "3cf5efc43834fe06a63ae5f2957fc079"
server_key_version: "encryptionkey@0"
I20260808 15:47:30.778970 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:30.802546 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:30.807044 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:30.807344 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:30.826284 14986 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34779
I20260808 15:47:30.826329 15060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34779 every 8 connection(s)
I20260808 15:47:30.827729 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:30.830015 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 14986
I20260808 15:47:30.830209 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:30.830811 24072 external_mini_cluster.cc:1428] Setting key 1e60e20ec8c7695189cd731f7b82cab0
I20260808 15:47:30.832211 15061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:47:30 dist-test-slave-gp6n krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:47:30.846554 15061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc: Bootstrap starting.
I20260808 15:47:30.850010 15061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:30.851106 15061 log.cc:824] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:30.853935 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:30.832669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:51117 (local address 127.0.0.1:34779)
0808 15:47:30.833426 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:30.833437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:30.833478 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:47:30.835167 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:30.835188 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:30.835578 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:30.835913 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:30.837746 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:30.839185 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:30.840984 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:30.841655 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:30.841786 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:30.846713 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:30.846725 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:30.846751 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:30.846788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:30.849755 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:30.850659 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:30.850665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:30.850671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:30.850830 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:30.851305 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:30.851315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:30.851317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:30.851839 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:30.852191 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:30.852680 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 15:47:30.852978 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":348,"threads_started":1}
I20260808 15:47:30.854087 15061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc: No bootstrap required, opened a new log
I20260808 15:47:30.857867 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:30.862264 15061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260808 15:47:30.862722 15061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:30.862845 15061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498180e1b6174336a04e7e78064f0bcc, State: Initialized, Role: FOLLOWER
I20260808 15:47:30.863492 15061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260808 15:47:30.863756 15061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:30.863906 15061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:30.864111 15061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:30.866969 15061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260808 15:47:30.867515 15061 leader_election.cc:302] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 498180e1b6174336a04e7e78064f0bcc; no voters:
I20260808 15:47:30.868160 15061 leader_election.cc:288] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:30.868278 15073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:30.870548 15073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [term 1 LEADER]: Becoming Leader. State: Replica: 498180e1b6174336a04e7e78064f0bcc, State: Running, Role: LEADER
I20260808 15:47:30.871232 15073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260808 15:47:30.871609 15061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:30.874512 15074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "498180e1b6174336a04e7e78064f0bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } } }
I20260808 15:47:30.874913 15074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:30.876349 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 498180e1b6174336a04e7e78064f0bcc. Latest consensus state: current_term: 1 leader_uuid: "498180e1b6174336a04e7e78064f0bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498180e1b6174336a04e7e78064f0bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } } }
I20260808 15:47:30.876700 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:30.880234 15079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:30.880765 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:30.862318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:35131 (local address 127.0.0.1:34779)
0808 15:47:30.862532 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:30.862538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:30.862995 (+ 457us) server_negotiation.cc:407] Connection header received
0808 15:47:30.863212 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:30.863217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:30.863290 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:30.863394 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:30.864914 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:30.866072 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:30.867387 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:30.868124 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:30.868280 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:30.871011 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:30.871019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:30.871023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:30.871063 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:30.875643 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:30.876866 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:30.876876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:30.876882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:30.877163 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:30.877905 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:30.877913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:30.877947 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:30.878457 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:30.878847 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:30.880146 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0808 15:47:30.880369 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:47:30.887138 15079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:30.895838 15079 catalog_manager.cc:1347] Generated new cluster ID: bc07177d938244c384f1637df606d023
I20260808 15:47:30.895968 15079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:30.923478 15079 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:30.924793 15079 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:30.934815 15079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 498180e1b6174336a04e7e78064f0bcc: Generated new TSK 0
I20260808 15:47:30.935846 15079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:30.948124 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 14986
I20260808 15:47:30.949250 15056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:31.043041 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 14986
W20260808 15:47:31.064687 14970 connection.cc:535] client connection to 127.0.0.1:34779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34779): Connection reset by peer (error 104)
2026-08-08T15:47:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:31 dist-test-slave-gp6n krb5kdc[15098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:31 dist-test-slave-gp6n krb5kdc[15098](info): set up 2 sockets
Aug 08 15:47:31 dist-test-slave-gp6n krb5kdc[15098](info): commencing operation
krb5kdc: starting...
W20260808 15:47:32.231312 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:32 dist-test-slave-gp6n krb5kdc[15098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:32.558386 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:32.900468 15112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:32.900764 15112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:32.900857 15112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:32.908305 15112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:32.908424 15112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:32.908470 15112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:32.908533 15112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:32.908577 15112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:32.918388 15112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:32.919683 15112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:32.920920 15112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:32.932768 15117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:32.932785 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:32.932789 15120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:32.934090 15112 server_base.cc:1034] running on GCE node
I20260808 15:47:32.934944 15112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:32.936375 15112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:32.937602 15112 hybrid_clock.cc:648] HybridClock initialized: now 1786204052937549 us; error 53 us; skew 500 ppm
I20260808 15:47:32.937958 15112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:32 dist-test-slave-gp6n krb5kdc[15098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:32.941334 15112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:32.942960 15112 webserver.cc:460] Webserver started at http://127.0.0.1:35941/ using document root <none> and password file <none>
I20260808 15:47:32.943656 15112 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:32.943811 15112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:32.944124 15112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:32.947203 15112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "3109d78ab87043b896c6bf2267f889ec"
format_stamp: "Formatted at 2026-08-08 15:47:32 on dist-test-slave-gp6n"
server_key: "16dededfbe3c05076eb38dad9e0840d3"
server_key_iv: "12531407e10e916765a03f8173e29cae"
server_key_version: "encryptionkey@0"
I20260808 15:47:32.947990 15112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "3109d78ab87043b896c6bf2267f889ec"
format_stamp: "Formatted at 2026-08-08 15:47:32 on dist-test-slave-gp6n"
server_key: "16dededfbe3c05076eb38dad9e0840d3"
server_key_iv: "12531407e10e916765a03f8173e29cae"
server_key_version: "encryptionkey@0"
I20260808 15:47:32.953228 15112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:47:32.956712 15127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:32.958024 15112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:47:32.958210 15112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "3109d78ab87043b896c6bf2267f889ec"
format_stamp: "Formatted at 2026-08-08 15:47:32 on dist-test-slave-gp6n"
server_key: "16dededfbe3c05076eb38dad9e0840d3"
server_key_iv: "12531407e10e916765a03f8173e29cae"
server_key_version: "encryptionkey@0"
I20260808 15:47:32.958482 15112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:32.979286 15112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:32.983872 15112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:32.984167 15112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:33.002700 15112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36551
I20260808 15:47:33.002720 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36551 every 8 connection(s)
I20260808 15:47:33.004069 15112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:33.007870 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15112
I20260808 15:47:33.007962 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:33.008061 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:33.008637 24072 external_mini_cluster.cc:1428] Setting key 3cf4f4f594162f2d4499a787b4226af9
I20260808 15:47:33.021893 15182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec: Bootstrap starting.
I20260808 15:47:33.022110 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:33.010562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:58763 (local address 127.0.0.1:36551)
0808 15:47:33.011426 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:33.011441 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:33.011484 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:47:33.013088 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:33.013111 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:33.013513 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:33.013842 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:33.014513 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:33.015857 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:33.017666 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:33.018337 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:33.018542 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:33.019844 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:33.019851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:33.019864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:33.020404 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:33.020476 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:33.020503 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:33.020617 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:33.020905 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 15:47:33.021117 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":412,"threads_started":1}
I20260808 15:47:33.025208 15182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:33.025857 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:33.026477 15182 log.cc:824] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:33.029783 15182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec: No bootstrap required, opened a new log
I20260808 15:47:33.037818 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:33.030858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:40483 (local address 127.0.0.1:36551)
0808 15:47:33.031025 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:33.031032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:33.031661 (+ 629us) server_negotiation.cc:407] Connection header received
0808 15:47:33.031779 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:33.031785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:33.031859 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:33.031967 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:33.032456 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:33.033630 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:33.035033 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:33.035787 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:33.035914 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:33.036972 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:33.036979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:33.036984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:33.037005 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:33.037076 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:33.037097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:33.037205 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:33.037398 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 15:47:33.037459 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:47:33.039408 15182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260808 15:47:33.039752 15182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:33.039826 15182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3109d78ab87043b896c6bf2267f889ec, State: Initialized, Role: FOLLOWER
I20260808 15:47:33.040318 15182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260808 15:47:33.040525 15182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:33.040621 15182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:33.040746 15182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:33.042881 15182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260808 15:47:33.043334 15182 leader_election.cc:302] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3109d78ab87043b896c6bf2267f889ec; no voters:
I20260808 15:47:33.043901 15182 leader_election.cc:288] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:33.044070 15193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:33.045811 15193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [term 1 LEADER]: Becoming Leader. State: Replica: 3109d78ab87043b896c6bf2267f889ec, State: Running, Role: LEADER
I20260808 15:47:33.046293 15182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:33.046452 15193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260808 15:47:33.049526 15195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3109d78ab87043b896c6bf2267f889ec. Latest consensus state: current_term: 1 leader_uuid: "3109d78ab87043b896c6bf2267f889ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260808 15:47:33.049962 15195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:33.050503 15194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3109d78ab87043b896c6bf2267f889ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3109d78ab87043b896c6bf2267f889ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260808 15:47:33.050796 15194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:33.052047 15202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:33.056922 15202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:33.065503 15202 catalog_manager.cc:1347] Generated new cluster ID: ecd835e228654889a38acca364f3a84e
I20260808 15:47:33.065618 15202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:33.081112 15202 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:33.083009 15202 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:33.095300 15202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3109d78ab87043b896c6bf2267f889ec: Generated new TSK 0
I20260808 15:47:33.096357 15202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:33.102744 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15112
W20260808 15:47:33.104461 15177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:58763
W20260808 15:47:33.105297 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:33.105472 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15112
W20260808 15:47:33.127004 15096 connection.cc:535] client connection to 127.0.0.1:36551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36551): Connection reset by peer (error 104)
2026-08-08T15:47:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:33 dist-test-slave-gp6n krb5kdc[15217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:33 dist-test-slave-gp6n krb5kdc[15217](info): set up 2 sockets
Aug 08 15:47:33 dist-test-slave-gp6n krb5kdc[15217](info): commencing operation
krb5kdc: starting...
W20260808 15:47:34.310736 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.006s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:34 dist-test-slave-gp6n krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:34.634567 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:34.979167 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:34.979552 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:34.979719 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:34.986984 15231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:34.987147 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:34.987252 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:34.987360 15231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:34.987437 15231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:34.997360 15231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:34.998807 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:35.000190 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:35.012632 15239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:35.012723 15236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:35.012712 15237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:35.013742 15231 server_base.cc:1034] running on GCE node
I20260808 15:47:35.014565 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:35.015954 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:35.017184 15231 hybrid_clock.cc:648] HybridClock initialized: now 1786204055017139 us; error 39 us; skew 500 ppm
I20260808 15:47:35.017580 15231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:35 dist-test-slave-gp6n krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:35.020782 15231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:35.022300 15231 webserver.cc:460] Webserver started at http://127.0.0.1:41829/ using document root <none> and password file <none>
I20260808 15:47:35.023012 15231 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:35.023165 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:35.023475 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:35.026532 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "a8887006ceac4bed8499ade65a7124ff"
format_stamp: "Formatted at 2026-08-08 15:47:35 on dist-test-slave-gp6n"
server_key: "757d2af3eb2a5eeecc0df996234a3882"
server_key_iv: "c758455946f624784d375358de6453d9"
server_key_version: "encryptionkey@0"
I20260808 15:47:35.027356 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "a8887006ceac4bed8499ade65a7124ff"
format_stamp: "Formatted at 2026-08-08 15:47:35 on dist-test-slave-gp6n"
server_key: "757d2af3eb2a5eeecc0df996234a3882"
server_key_iv: "c758455946f624784d375358de6453d9"
server_key_version: "encryptionkey@0"
I20260808 15:47:35.032753 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:35.036418 15248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:35.037748 15231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:47:35.037938 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "a8887006ceac4bed8499ade65a7124ff"
format_stamp: "Formatted at 2026-08-08 15:47:35 on dist-test-slave-gp6n"
server_key: "757d2af3eb2a5eeecc0df996234a3882"
server_key_iv: "c758455946f624784d375358de6453d9"
server_key_version: "encryptionkey@0"
I20260808 15:47:35.038153 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:35.044198 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:35.048506 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:35.048826 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:35.068239 15231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36265
I20260808 15:47:35.068290 15303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36265 every 8 connection(s)
I20260808 15:47:35.069715 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:35.073479 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:35.074985 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15231
I20260808 15:47:35.075169 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:35.075778 24072 external_mini_cluster.cc:1428] Setting key 5f5700d9c10074c4e627d3bc096012a8
I20260808 15:47:35.083315 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:35.077560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:35091 (local address 127.0.0.1:36265)
0808 15:47:35.078395 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:35.078453 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:35.078509 (+ 56us) server_negotiation.cc:407] Connection header received
0808 15:47:35.080111 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:35.080133 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:35.080473 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:35.080793 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:35.081810 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:35.081816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:35.081829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:35.081842 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:35.081901 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:35.081922 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:35.082094 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:35.082198 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 15:47:35.082409 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":391,"threads_started":1}
I20260808 15:47:35.086947 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:35.086961 15304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff: Bootstrap starting.
I20260808 15:47:35.090771 15304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:35.092146 15304 log.cc:824] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:35.094980 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:35.092345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:37375 (local address 127.0.0.1:36265)
0808 15:47:35.092525 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:35.092530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:35.093033 (+ 503us) server_negotiation.cc:407] Connection header received
0808 15:47:35.093224 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:35.093236 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:35.093279 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:35.093381 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:35.094302 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:35.094307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:35.094312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:35.094329 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:35.094376 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:35.094390 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:35.094543 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:35.094648 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 15:47:35.094709 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:47:35.095045 15304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff: No bootstrap required, opened a new log
I20260808 15:47:35.102936 15304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } }
I20260808 15:47:35.103320 15304 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:35.103431 15304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8887006ceac4bed8499ade65a7124ff, State: Initialized, Role: FOLLOWER
I20260808 15:47:35.104067 15304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } }
I20260808 15:47:35.104346 15304 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:35.104511 15304 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:35.104687 15304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:35.106935 15304 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } }
I20260808 15:47:35.107422 15304 leader_election.cc:302] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8887006ceac4bed8499ade65a7124ff; no voters:
I20260808 15:47:35.107975 15304 leader_election.cc:288] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:35.108104 15314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:35.109475 15314 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [term 1 LEADER]: Becoming Leader. State: Replica: a8887006ceac4bed8499ade65a7124ff, State: Running, Role: LEADER
I20260808 15:47:35.110100 15314 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } }
I20260808 15:47:35.110261 15304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:35.112916 15316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8887006ceac4bed8499ade65a7124ff. Latest consensus state: current_term: 1 leader_uuid: "a8887006ceac4bed8499ade65a7124ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } } }
I20260808 15:47:35.113324 15316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:35.113834 15315 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8887006ceac4bed8499ade65a7124ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8887006ceac4bed8499ade65a7124ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36265 } } }
I20260808 15:47:35.114146 15315 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:35.115100 15322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:35.120712 15322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:35.129293 15322 catalog_manager.cc:1347] Generated new cluster ID: 53086b61a126412fba33aeb3ef3529b4
I20260808 15:47:35.129417 15322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:35.148696 15322 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:35.149135 15322 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:35.164207 15322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8887006ceac4bed8499ade65a7124ff: Generated new TSK 0
I20260808 15:47:35.165078 15322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:35.173784 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15231
W20260808 15:47:35.174980 15299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:35091
W20260808 15:47:35.175669 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:35.175814 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15231
2026-08-08T15:47:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:35 dist-test-slave-gp6n krb5kdc[15336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:35 dist-test-slave-gp6n krb5kdc[15336](info): set up 2 sockets
Aug 08 15:47:35 dist-test-slave-gp6n krb5kdc[15336](info): commencing operation
krb5kdc: starting...
W20260808 15:47:36.361726 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:36 dist-test-slave-gp6n krb5kdc[15336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:36.677217 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:37.033037 15351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:37.033329 15351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:37.033432 15351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:37.040694 15351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:37.040812 15351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:37.040859 15351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:37.040915 15351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:37.040961 15351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:37.051249 15351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:37.052544 15351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:37.053897 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:37.066367 15357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:37.066465 15361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:37.066443 15359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:37.067272 15351 server_base.cc:1034] running on GCE node
I20260808 15:47:37.068133 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:37.069581 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:37.070814 15351 hybrid_clock.cc:648] HybridClock initialized: now 1786204057070770 us; error 44 us; skew 500 ppm
I20260808 15:47:37.071177 15351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:37 dist-test-slave-gp6n krb5kdc[15336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:37.074461 15351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:37.075973 15351 webserver.cc:460] Webserver started at http://127.0.0.1:34169/ using document root <none> and password file <none>
I20260808 15:47:37.076683 15351 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:37.076835 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:37.077147 15351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:37.080188 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "48d8ce62c69c4fe98d3af14899a5f948"
format_stamp: "Formatted at 2026-08-08 15:47:37 on dist-test-slave-gp6n"
server_key: "84f81b7e69e052105410fae4a86f1215"
server_key_iv: "ecd7012a517a8a9469b7ee060689a66a"
server_key_version: "encryptionkey@0"
I20260808 15:47:37.080963 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "48d8ce62c69c4fe98d3af14899a5f948"
format_stamp: "Formatted at 2026-08-08 15:47:37 on dist-test-slave-gp6n"
server_key: "84f81b7e69e052105410fae4a86f1215"
server_key_iv: "ecd7012a517a8a9469b7ee060689a66a"
server_key_version: "encryptionkey@0"
I20260808 15:47:37.086180 15351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 15:47:37.089551 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:37.090875 15351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:37.091066 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "48d8ce62c69c4fe98d3af14899a5f948"
format_stamp: "Formatted at 2026-08-08 15:47:37 on dist-test-slave-gp6n"
server_key: "84f81b7e69e052105410fae4a86f1215"
server_key_iv: "ecd7012a517a8a9469b7ee060689a66a"
server_key_version: "encryptionkey@0"
I20260808 15:47:37.091264 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:37.109508 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:37.113684 15351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:37.113977 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:37.133574 15351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32791
I20260808 15:47:37.133595 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32791 every 8 connection(s)
I20260808 15:47:37.135102 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:37.138615 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15351
I20260808 15:47:37.138826 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:37.139215 15426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:37.139415 24072 external_mini_cluster.cc:1428] Setting key aed2315443ca783a7e3ad0ce8245383f
I20260808 15:47:37.151743 15426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948: Bootstrap starting.
Aug 08 15:47:37 dist-test-slave-gp6n krb5kdc[15336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:47:37.155606 15426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:37.156924 15426 log.cc:824] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:37.160487 15426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948: No bootstrap required, opened a new log
I20260808 15:47:37.163074 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:37.141278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:53873 (local address 127.0.0.1:32791)
0808 15:47:37.142052 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:37.142064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:37.142109 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:47:37.144067 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:37.144087 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:37.144470 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:37.144876 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:37.146354 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:37.147656 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:37.148932 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:37.149686 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:37.149889 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:37.154731 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:37.154739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:37.154754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:37.154797 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:37.158515 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:37.159549 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:37.159555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:37.159561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:37.159727 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:37.160245 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:37.160248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:37.160250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:37.160824 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:37.161076 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:37.161625 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 15:47:37.161924 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":336,"threads_started":1}
I20260808 15:47:37.167196 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:37.170012 15426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260808 15:47:37.170578 15426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:37.170725 15426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d8ce62c69c4fe98d3af14899a5f948, State: Initialized, Role: FOLLOWER
I20260808 15:47:37.171522 15426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260808 15:47:37.171850 15426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:37.172009 15426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:37.172210 15426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:37.175860 15426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260808 15:47:37.176518 15426 leader_election.cc:302] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48d8ce62c69c4fe98d3af14899a5f948; no voters:
I20260808 15:47:37.177224 15426 leader_election.cc:288] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:37.177349 15437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:37.179088 15437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [term 1 LEADER]: Becoming Leader. State: Replica: 48d8ce62c69c4fe98d3af14899a5f948, State: Running, Role: LEADER
I20260808 15:47:37.179790 15437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260808 15:47:37.179986 15426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:37.189162 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:37.173027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:59671 (local address 127.0.0.1:32791)
0808 15:47:37.173285 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:37.173291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:37.173670 (+ 379us) server_negotiation.cc:407] Connection header received
0808 15:47:37.173853 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:37.173857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:37.173923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:37.174027 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:37.176132 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:37.177017 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:37.178847 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:37.179547 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:37.179708 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:37.182816 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:37.182821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:37.182824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:37.182855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:37.185917 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:37.186847 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:37.186852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:37.186854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:37.186971 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:37.187438 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:37.187443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:37.187447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:37.187790 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:37.188085 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:37.188504 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:47:37.188676 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:47:37.190141 15439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48d8ce62c69c4fe98d3af14899a5f948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260808 15:47:37.193298 15439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:37.193886 15440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48d8ce62c69c4fe98d3af14899a5f948. Latest consensus state: current_term: 1 leader_uuid: "48d8ce62c69c4fe98d3af14899a5f948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d8ce62c69c4fe98d3af14899a5f948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260808 15:47:37.194433 15440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:37.195971 15453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:37.202507 15453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:37.211639 15453 catalog_manager.cc:1347] Generated new cluster ID: 73f6987b2ae246e98c1d693200b20548
I20260808 15:47:37.211759 15453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:37.219621 15453 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:37.221472 15453 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:37.231601 15453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48d8ce62c69c4fe98d3af14899a5f948: Generated new TSK 0
I20260808 15:47:37.232414 15453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:37.250792 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15351
I20260808 15:47:37.251940 15421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:37.345242 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15351
W20260808 15:47:37.366168 15334 connection.cc:535] client connection to 127.0.0.1:32791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32791): Connection reset by peer (error 104)
2026-08-08T15:47:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:37 dist-test-slave-gp6n krb5kdc[15462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:37 dist-test-slave-gp6n krb5kdc[15462](info): set up 2 sockets
Aug 08 15:47:37 dist-test-slave-gp6n krb5kdc[15462](info): commencing operation
krb5kdc: starting...
W20260808 15:47:38.534116 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:38 dist-test-slave-gp6n krb5kdc[15462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:38.865792 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:39.227464 15475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:39.227818 15475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:39.227980 15475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:39.236159 15475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:39.236304 15475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:39.236413 15475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:39.236523 15475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:39.236614 15475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:39.247021 15475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:39.248385 15475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:39.249768 15475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:39.262089 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:39.262128 15480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:39.262164 15481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:39.263041 15475 server_base.cc:1034] running on GCE node
I20260808 15:47:39.263788 15475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:39.265141 15475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:39.266382 15475 hybrid_clock.cc:648] HybridClock initialized: now 1786204059266343 us; error 44 us; skew 500 ppm
I20260808 15:47:39.266774 15475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:39 dist-test-slave-gp6n krb5kdc[15462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:39.269913 15475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:39.271407 15475 webserver.cc:460] Webserver started at http://127.0.0.1:44383/ using document root <none> and password file <none>
I20260808 15:47:39.272080 15475 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:39.272224 15475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:39.272540 15475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:39.275722 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "711f41ee8bfa4cff8e49e365e5c674ce"
format_stamp: "Formatted at 2026-08-08 15:47:39 on dist-test-slave-gp6n"
server_key: "48818b2b915bfbc19ea6ebf8abb26699"
server_key_iv: "bb0f47301128c2ee8a6d94984115793e"
server_key_version: "encryptionkey@0"
I20260808 15:47:39.276518 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "711f41ee8bfa4cff8e49e365e5c674ce"
format_stamp: "Formatted at 2026-08-08 15:47:39 on dist-test-slave-gp6n"
server_key: "48818b2b915bfbc19ea6ebf8abb26699"
server_key_iv: "bb0f47301128c2ee8a6d94984115793e"
server_key_version: "encryptionkey@0"
I20260808 15:47:39.281647 15475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:47:39.285168 15492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:39.286625 15475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:39.286791 15475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "711f41ee8bfa4cff8e49e365e5c674ce"
format_stamp: "Formatted at 2026-08-08 15:47:39 on dist-test-slave-gp6n"
server_key: "48818b2b915bfbc19ea6ebf8abb26699"
server_key_iv: "bb0f47301128c2ee8a6d94984115793e"
server_key_version: "encryptionkey@0"
I20260808 15:47:39.287197 15475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:39.308544 15475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:39.313371 15475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:39.313679 15475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:39.334733 15475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36947
I20260808 15:47:39.334758 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36947 every 8 connection(s)
I20260808 15:47:39.336212 15475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:39.337918 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15475
I20260808 15:47:39.338126 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:39.338681 24072 external_mini_cluster.cc:1428] Setting key 62aba101bb71d1ebb48cc1d281984cb3
I20260808 15:47:39.340638 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:39.352995 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:39.340532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:57685 (local address 127.0.0.1:36947)
0808 15:47:39.341403 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:39.341416 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:39.341458 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:47:39.343215 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:39.343252 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:39.343679 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:39.344028 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:39.344657 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:39.346266 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:39.348262 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:39.348982 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:39.349118 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:39.350608 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:39.350616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:39.350629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:39.351225 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:39.351304 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:39.351330 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:39.351472 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:39.351797 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 15:47:39.352005 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":423,"threads_started":1}
I20260808 15:47:39.353755 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce: Bootstrap starting.
I20260808 15:47:39.356985 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:39.357326 15549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:39.358328 15549 log.cc:824] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:39.361359 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce: No bootstrap required, opened a new log
I20260808 15:47:39.368136 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:39.361886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:40465 (local address 127.0.0.1:36947)
0808 15:47:39.362078 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:39.362086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:39.362461 (+ 375us) server_negotiation.cc:407] Connection header received
0808 15:47:39.362683 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:39.362689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:39.362760 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:39.362862 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:39.363412 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:39.364242 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:39.365548 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:39.366056 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:39.366202 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:39.367379 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:39.367384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:39.367387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:39.367405 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:39.367445 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:39.367461 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:39.367594 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:39.367782 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 15:47:39.367840 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 15:47:39.369860 15549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260808 15:47:39.370251 15549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:39.370360 15549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711f41ee8bfa4cff8e49e365e5c674ce, State: Initialized, Role: FOLLOWER
I20260808 15:47:39.371011 15549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260808 15:47:39.371250 15549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:39.371397 15549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:39.371568 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:39.373967 15549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260808 15:47:39.374531 15549 leader_election.cc:302] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 711f41ee8bfa4cff8e49e365e5c674ce; no voters:
I20260808 15:47:39.375157 15549 leader_election.cc:288] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:39.375320 15559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:39.376988 15559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [term 1 LEADER]: Becoming Leader. State: Replica: 711f41ee8bfa4cff8e49e365e5c674ce, State: Running, Role: LEADER
I20260808 15:47:39.377549 15549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:39.377621 15559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260808 15:47:39.381417 15561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 711f41ee8bfa4cff8e49e365e5c674ce. Latest consensus state: current_term: 1 leader_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260808 15:47:39.381899 15561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:39.382799 15560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711f41ee8bfa4cff8e49e365e5c674ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260808 15:47:39.383276 15560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:39.387585 15567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:39.393558 15567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:39.402791 15567 catalog_manager.cc:1347] Generated new cluster ID: 88397805155c425e9e40bfc1cdc0b678
I20260808 15:47:39.402937 15567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:39.425932 15567 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:39.427866 15567 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:39.438876 15567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 711f41ee8bfa4cff8e49e365e5c674ce: Generated new TSK 0
I20260808 15:47:39.439771 15567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:39.452862 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15475
W20260808 15:47:39.454489 15544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:57685
W20260808 15:47:39.455325 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15475: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:39.455477 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15475
W20260808 15:47:39.477073 15460 connection.cc:535] client connection to 127.0.0.1:36947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36947): Connection reset by peer (error 104)
2026-08-08T15:47:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:39 dist-test-slave-gp6n krb5kdc[15586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:39 dist-test-slave-gp6n krb5kdc[15586](info): set up 2 sockets
Aug 08 15:47:39 dist-test-slave-gp6n krb5kdc[15586](info): commencing operation
krb5kdc: starting...
W20260808 15:47:40.649164 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:40 dist-test-slave-gp6n krb5kdc[15586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:47:40.969260 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:41.320866 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:41.321177 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:41.321357 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:41.328809 15599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:41.328960 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:41.329049 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:41.329162 15599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:41.329239 15599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:41.339440 15599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:41.340865 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:41.342252 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:41.354676 15605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:41.354676 15608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:41.354712 15606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:41.355939 15599 server_base.cc:1034] running on GCE node
I20260808 15:47:41.356756 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:41.358228 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:41.359467 15599 hybrid_clock.cc:648] HybridClock initialized: now 1786204061359415 us; error 40 us; skew 500 ppm
I20260808 15:47:41.359843 15599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:41 dist-test-slave-gp6n krb5kdc[15586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:41.363168 15599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:47:41.364838 15599 webserver.cc:460] Webserver started at http://127.0.0.1:46781/ using document root <none> and password file <none>
I20260808 15:47:41.365561 15599 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:41.365710 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:41.366042 15599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:41.369369 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "91942c7c5f36453490d624bdea72eca3"
format_stamp: "Formatted at 2026-08-08 15:47:41 on dist-test-slave-gp6n"
server_key: "754ea55f546e687b1994ed219603b23d"
server_key_iv: "c9cb4f93daaa494b3b025bce46fe7bb5"
server_key_version: "encryptionkey@0"
I20260808 15:47:41.370226 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "91942c7c5f36453490d624bdea72eca3"
format_stamp: "Formatted at 2026-08-08 15:47:41 on dist-test-slave-gp6n"
server_key: "754ea55f546e687b1994ed219603b23d"
server_key_iv: "c9cb4f93daaa494b3b025bce46fe7bb5"
server_key_version: "encryptionkey@0"
I20260808 15:47:41.375845 15599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:47:41.379701 15616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:41.381053 15599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260808 15:47:41.381246 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "91942c7c5f36453490d624bdea72eca3"
format_stamp: "Formatted at 2026-08-08 15:47:41 on dist-test-slave-gp6n"
server_key: "754ea55f546e687b1994ed219603b23d"
server_key_iv: "c9cb4f93daaa494b3b025bce46fe7bb5"
server_key_version: "encryptionkey@0"
I20260808 15:47:41.381490 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:41.388439 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:41.392654 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:41.392953 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:41.411213 15599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34851
I20260808 15:47:41.411266 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34851 every 8 connection(s)
I20260808 15:47:41.412621 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:41.416271 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:41.420517 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15599
I20260808 15:47:41.420720 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:41.421312 24072 external_mini_cluster.cc:1428] Setting key 5f648f757e44425133bec70bbc299817
I20260808 15:47:41.427778 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3: Bootstrap starting.
I20260808 15:47:41.429571 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:41.423121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:58897 (local address 127.0.0.1:34851)
0808 15:47:41.423861 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:41.423872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:41.423913 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:47:41.426197 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:41.426218 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:41.426617 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:41.426934 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:41.428021 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:41.428027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:41.428039 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:41.428054 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:41.428113 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:41.428132 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:41.428308 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:41.428451 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 15:47:41.428649 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":330,"threads_started":1}
I20260808 15:47:41.431092 15672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:41.432253 15672 log.cc:824] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:41.433151 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:41.435413 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3: No bootstrap required, opened a new log
I20260808 15:47:41.440915 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:41.438333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:55873 (local address 127.0.0.1:34851)
0808 15:47:41.438530 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:41.438536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:41.439029 (+ 493us) server_negotiation.cc:407] Connection header received
0808 15:47:41.439159 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:41.439169 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:41.439232 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:41.439359 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:41.440240 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:41.440247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:41.440257 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:41.440275 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:41.440325 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:41.440345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:41.440478 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:41.440532 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 15:47:41.440592 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:47:41.444453 15672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260808 15:47:41.444890 15672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:41.445021 15672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942c7c5f36453490d624bdea72eca3, State: Initialized, Role: FOLLOWER
I20260808 15:47:41.445667 15672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260808 15:47:41.445916 15672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:41.446056 15672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:41.446233 15672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:41.448436 15672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260808 15:47:41.448921 15672 leader_election.cc:302] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91942c7c5f36453490d624bdea72eca3; no voters:
I20260808 15:47:41.449571 15672 leader_election.cc:288] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:41.449697 15682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:41.450377 15682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [term 1 LEADER]: Becoming Leader. State: Replica: 91942c7c5f36453490d624bdea72eca3, State: Running, Role: LEADER
I20260808 15:47:41.450976 15682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260808 15:47:41.451705 15672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:41.454231 15683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91942c7c5f36453490d624bdea72eca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260808 15:47:41.454293 15684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91942c7c5f36453490d624bdea72eca3. Latest consensus state: current_term: 1 leader_uuid: "91942c7c5f36453490d624bdea72eca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91942c7c5f36453490d624bdea72eca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260808 15:47:41.454710 15683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:41.454730 15684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:41.456668 15690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:41.461840 15690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:41.471202 15690 catalog_manager.cc:1347] Generated new cluster ID: 4f635062e6354aacbaecee438bf7e8d3
I20260808 15:47:41.471336 15690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:41.514648 15690 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:41.515108 15690 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:41.529613 15690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91942c7c5f36453490d624bdea72eca3: Generated new TSK 0
I20260808 15:47:41.530503 15690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:41.540769 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15599
W20260808 15:47:41.541975 15667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:58897
W20260808 15:47:41.542706 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:41.542848 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15599
2026-08-08T15:47:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:41 dist-test-slave-gp6n krb5kdc[15703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:41 dist-test-slave-gp6n krb5kdc[15703](info): set up 2 sockets
Aug 08 15:47:41 dist-test-slave-gp6n krb5kdc[15703](info): commencing operation
krb5kdc: starting...
W20260808 15:47:42.728633 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:42 dist-test-slave-gp6n krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:43.061668 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33397
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:33397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:43.407866 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:43.408211 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:43.408370 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:43.415620 15717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:43.415768 15717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:43.415881 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:43.415987 15717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:43.416061 15717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:43.425665 15717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:33397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:43.426972 15717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:43.428269 15717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:43.440136 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:43.440232 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:43.440205 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:43.441031 15717 server_base.cc:1034] running on GCE node
I20260808 15:47:43.441783 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:43.443260 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:43.444490 15717 hybrid_clock.cc:648] HybridClock initialized: now 1786204063444457 us; error 34 us; skew 500 ppm
I20260808 15:47:43.444861 15717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:43 dist-test-slave-gp6n krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:43.454579 15717 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:43.456138 15717 webserver.cc:460] Webserver started at http://127.23.130.62:41219/ using document root <none> and password file <none>
I20260808 15:47:43.456840 15717 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:43.456977 15717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:43.457297 15717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:43.460341 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "db03732e599a4e3086c2316d3d6f518c"
format_stamp: "Formatted at 2026-08-08 15:47:43 on dist-test-slave-gp6n"
server_key: "8987f8e0295cb010b206ac130121ad86"
server_key_iv: "37dbe3cda3dc2b43af39023200f55016"
server_key_version: "encryptionkey@0"
I20260808 15:47:43.461135 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "db03732e599a4e3086c2316d3d6f518c"
format_stamp: "Formatted at 2026-08-08 15:47:43 on dist-test-slave-gp6n"
server_key: "8987f8e0295cb010b206ac130121ad86"
server_key_iv: "37dbe3cda3dc2b43af39023200f55016"
server_key_version: "encryptionkey@0"
I20260808 15:47:43.466281 15717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:47:43.469748 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:43.471150 15717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:43.471345 15717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "db03732e599a4e3086c2316d3d6f518c"
format_stamp: "Formatted at 2026-08-08 15:47:43 on dist-test-slave-gp6n"
server_key: "8987f8e0295cb010b206ac130121ad86"
server_key_iv: "37dbe3cda3dc2b43af39023200f55016"
server_key_version: "encryptionkey@0"
I20260808 15:47:43.471556 15717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:43.482786 15717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:43.486991 15717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:43.487268 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:43.506781 15717 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:33397
I20260808 15:47:43.506814 15792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:33397 every 8 connection(s)
I20260808 15:47:43.508176 15717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:43.512095 15793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:43.513445 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15717
I20260808 15:47:43.513639 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:43.514112 24072 external_mini_cluster.cc:1428] Setting key a3add2ca03769a3a982c86392b0b87ac
I20260808 15:47:43.527405 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c: Bootstrap starting.
Aug 08 15:47:43 dist-test-slave-gp6n krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:43.531488 15793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:43.532703 15793 log.cc:824] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:43.535838 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c: No bootstrap required, opened a new log
I20260808 15:47:43.538960 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:43.516062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34449 (local address 127.23.130.62:33397)
0808 15:47:43.516957 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:43.516968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:43.517009 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:47:43.518691 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:43.518717 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:43.519107 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:43.519440 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:43.521550 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:43.522975 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:43.525044 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:43.525757 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:43.525904 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:43.531277 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:43.531285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:43.531300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:43.531351 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:43.534629 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:43.535453 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:43.535460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:43.535466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:43.535634 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:43.536153 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:43.536162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:43.536164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:43.536699 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:43.537048 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:43.537583 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 15:47:43.537884 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":664,"thread_start_us":425,"threads_started":1}
I20260808 15:47:43.542948 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:43.544790 15793 raft_consensus.cc:348] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } }
I20260808 15:47:43.545226 15793 raft_consensus.cc:374] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:43.545361 15793 raft_consensus.cc:729] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db03732e599a4e3086c2316d3d6f518c, State: Initialized, Role: FOLLOWER
I20260808 15:47:43.546124 15793 consensus_queue.cc:260] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } }
I20260808 15:47:43.546482 15793 raft_consensus.cc:388] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:43.546655 15793 raft_consensus.cc:482] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:43.546849 15793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:43.550715 15793 raft_consensus.cc:504] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } }
I20260808 15:47:43.551267 15793 leader_election.cc:302] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db03732e599a4e3086c2316d3d6f518c; no voters:
I20260808 15:47:43.551977 15793 leader_election.cc:288] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:43.552042 15807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:43.554235 15793 sys_catalog.cc:564] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:43.554653 15807 raft_consensus.cc:686] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [term 1 LEADER]: Becoming Leader. State: Replica: db03732e599a4e3086c2316d3d6f518c, State: Running, Role: LEADER
I20260808 15:47:43.555315 15807 consensus_queue.cc:237] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } }
I20260808 15:47:43.563850 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [sys.catalog]: SysCatalogTable state changed. Reason: New leader db03732e599a4e3086c2316d3d6f518c. Latest consensus state: current_term: 1 leader_uuid: "db03732e599a4e3086c2316d3d6f518c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } } }
I20260808 15:47:43.564275 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:43.565294 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db03732e599a4e3086c2316d3d6f518c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db03732e599a4e3086c2316d3d6f518c" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 33397 } } }
I20260808 15:47:43.565640 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:43.569715 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:43.547376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38819 (local address 127.23.130.62:33397)
0808 15:47:43.547601 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:43.547610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:43.548110 (+ 500us) server_negotiation.cc:407] Connection header received
0808 15:47:43.548281 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:43.548284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:43.548353 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:43.548434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:43.550578 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:43.551466 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:43.553486 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:43.554542 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:43.554710 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:43.558760 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:43.558770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:43.558777 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:43.558828 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:43.564327 (+ 5499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:43.565629 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:43.565637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:43.565644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:43.565827 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:43.566622 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:43.566630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:43.566636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:43.567142 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:43.567477 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:43.568933 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0808 15:47:43.569182 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:47:43.574242 15820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:43.579402 15820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:43.589304 15820 catalog_manager.cc:1347] Generated new cluster ID: 404cb0a48c8c4c5ea6a8d156e65b7ef0
I20260808 15:47:43.589443 15820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:43.616171 15820 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:43.617556 15820 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:43.626094 15820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db03732e599a4e3086c2316d3d6f518c: Generated new TSK 0
I20260808 15:47:43.626978 15820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:43.633950 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15717
I20260808 15:47:43.635139 15788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:43.728952 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15717
W20260808 15:47:43.752228 15701 connection.cc:535] client connection to 127.23.130.62:33397 recv error: Network error: failed to read from TLS socket (remote: 127.23.130.62:33397): Connection reset by peer (error 104)
2026-08-08T15:47:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:43 dist-test-slave-gp6n krb5kdc[15830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:43 dist-test-slave-gp6n krb5kdc[15830](info): set up 2 sockets
Aug 08 15:47:43 dist-test-slave-gp6n krb5kdc[15830](info): commencing operation
krb5kdc: starting...
W20260808 15:47:44.976745 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:45 dist-test-slave-gp6n krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:45.298038 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:35439
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:45.656769 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:45.657050 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:45.657204 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:45.664888 15844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:45.665004 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:45.665108 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:45.665220 15844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:45.665303 15844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:45.675379 15844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:35439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:45.676847 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:45.678637 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:45.694489 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:45.694461 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:45.694684 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:45.695159 15844 server_base.cc:1034] running on GCE node
I20260808 15:47:45.696138 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:45.697583 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:45.698807 15844 hybrid_clock.cc:648] HybridClock initialized: now 1786204065698752 us; error 48 us; skew 500 ppm
I20260808 15:47:45.699172 15844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:45 dist-test-slave-gp6n krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:45.702266 15844 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:45.703714 15844 webserver.cc:460] Webserver started at http://127.23.130.62:41613/ using document root <none> and password file <none>
I20260808 15:47:45.704406 15844 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:45.704551 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:45.704859 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:45.708307 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "56048bbd69c74691a55039a8330c53d1"
format_stamp: "Formatted at 2026-08-08 15:47:45 on dist-test-slave-gp6n"
server_key: "494601b1549f23ac6e6cbe2a4c84cfe8"
server_key_iv: "904026ab47000824e3a6e8e6a5eacd11"
server_key_version: "encryptionkey@0"
I20260808 15:47:45.709079 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "56048bbd69c74691a55039a8330c53d1"
format_stamp: "Formatted at 2026-08-08 15:47:45 on dist-test-slave-gp6n"
server_key: "494601b1549f23ac6e6cbe2a4c84cfe8"
server_key_iv: "904026ab47000824e3a6e8e6a5eacd11"
server_key_version: "encryptionkey@0"
I20260808 15:47:45.714183 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:47:45.717772 15860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:45.719122 15844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:45.719308 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "56048bbd69c74691a55039a8330c53d1"
format_stamp: "Formatted at 2026-08-08 15:47:45 on dist-test-slave-gp6n"
server_key: "494601b1549f23ac6e6cbe2a4c84cfe8"
server_key_iv: "904026ab47000824e3a6e8e6a5eacd11"
server_key_version: "encryptionkey@0"
I20260808 15:47:45.719515 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:45.733953 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:45.738859 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:45.739140 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:45.758824 15844 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:35439
I20260808 15:47:45.758832 15916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:35439 every 8 connection(s)
I20260808 15:47:45.760280 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:45.760747 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15844
I20260808 15:47:45.760933 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:45.761605 24072 external_mini_cluster.cc:1428] Setting key 636c2b9b7eb509864446940066aee5c2
I20260808 15:47:45.764868 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:45.774724 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:45.763524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52363 (local address 127.23.130.62:35439)
0808 15:47:45.764298 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:45.764310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:45.764350 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:45.766058 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:45.766082 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:45.766526 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:45.766902 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:45.767482 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:45.768829 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:45.770196 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:45.770906 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:45.771111 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:45.772403 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:45.772410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:45.772424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:45.773010 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:45.773085 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:45.773112 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:45.773288 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:45.773574 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:47:45.773735 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":348,"threads_started":1}
I20260808 15:47:45.775712 15917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1: Bootstrap starting.
I20260808 15:47:45.778781 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:45.779479 15917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:45.780472 15917 log.cc:824] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:45.783910 15917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1: No bootstrap required, opened a new log
I20260808 15:47:45.789644 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:45.783339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53947 (local address 127.23.130.62:35439)
0808 15:47:45.783482 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:45.783486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:45.783949 (+ 463us) server_negotiation.cc:407] Connection header received
0808 15:47:45.784159 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:45.784163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:45.784219 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:45.784327 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:45.784828 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:45.785625 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:45.787018 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:45.787614 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:45.787765 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:45.788896 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:45.788901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:45.788905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:45.788921 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:45.788960 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:45.788978 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:45.789107 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:45.789309 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 15:47:45.789366 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 15:47:45.792212 15917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } }
I20260808 15:47:45.792591 15917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:45.792701 15917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56048bbd69c74691a55039a8330c53d1, State: Initialized, Role: FOLLOWER
I20260808 15:47:45.793252 15917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } }
I20260808 15:47:45.793520 15917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:45.793665 15917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:45.793850 15917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:45.796526 15917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } }
I20260808 15:47:45.797049 15917 leader_election.cc:302] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56048bbd69c74691a55039a8330c53d1; no voters:
I20260808 15:47:45.797605 15917 leader_election.cc:288] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:45.797736 15930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:45.799052 15930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [term 1 LEADER]: Becoming Leader. State: Replica: 56048bbd69c74691a55039a8330c53d1, State: Running, Role: LEADER
I20260808 15:47:45.799574 15930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } }
I20260808 15:47:45.799770 15917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:45.802680 15931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56048bbd69c74691a55039a8330c53d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } } }
I20260808 15:47:45.802659 15932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56048bbd69c74691a55039a8330c53d1. Latest consensus state: current_term: 1 leader_uuid: "56048bbd69c74691a55039a8330c53d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56048bbd69c74691a55039a8330c53d1" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 35439 } } }
I20260808 15:47:45.803100 15931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:45.803148 15932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:45.805505 15937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:45.810976 15937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:45.819716 15937 catalog_manager.cc:1347] Generated new cluster ID: 4aa3079065b74b9f9547901b5d84cd55
I20260808 15:47:45.819829 15937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:45.835058 15937 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:45.836345 15937 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:45.851285 15937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56048bbd69c74691a55039a8330c53d1: Generated new TSK 0
I20260808 15:47:45.852181 15937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:45.866070 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15844
W20260808 15:47:45.867406 15912 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52363
W20260808 15:47:45.868191 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15844: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:45.868321 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15844
W20260808 15:47:45.889446 15828 connection.cc:535] client connection to 127.23.130.62:35439 recv error: Network error: failed to read from TLS socket (remote: 127.23.130.62:35439): Connection reset by peer (error 104)
2026-08-08T15:47:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:45 dist-test-slave-gp6n krb5kdc[15953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:45 dist-test-slave-gp6n krb5kdc[15953](info): set up 2 sockets
Aug 08 15:47:45 dist-test-slave-gp6n krb5kdc[15953](info): commencing operation
krb5kdc: starting...
W20260808 15:47:47.073333 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:47 dist-test-slave-gp6n krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:47.399447 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39303
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:47.742239 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:47.742591 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:47.742784 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:47.750109 15966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:47.750236 15966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:47.750344 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:47.750449 15966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:47.750540 15966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:47.760293 15966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:39303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:47.761579 15966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:47.762923 15966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:47.774870 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:47.774874 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:47.774897 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:47.775636 15966 server_base.cc:1034] running on GCE node
I20260808 15:47:47.776443 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:47.777876 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:47.779122 15966 hybrid_clock.cc:648] HybridClock initialized: now 1786204067779079 us; error 57 us; skew 500 ppm
I20260808 15:47:47.779500 15966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:47 dist-test-slave-gp6n krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:47.782775 15966 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:47.784353 15966 webserver.cc:460] Webserver started at http://127.23.130.62:34135/ using document root <none> and password file <none>
I20260808 15:47:47.785053 15966 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:47.785205 15966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:47.785521 15966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:47.788578 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "23a30485e9c341a7bdf5bdc9d6ec1158"
format_stamp: "Formatted at 2026-08-08 15:47:47 on dist-test-slave-gp6n"
server_key: "d01e95b011b0543d12ae7bb1e9d62c34"
server_key_iv: "0c492a9c634cb668b587aab0c5e974de"
server_key_version: "encryptionkey@0"
I20260808 15:47:47.789357 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "23a30485e9c341a7bdf5bdc9d6ec1158"
format_stamp: "Formatted at 2026-08-08 15:47:47 on dist-test-slave-gp6n"
server_key: "d01e95b011b0543d12ae7bb1e9d62c34"
server_key_iv: "0c492a9c634cb668b587aab0c5e974de"
server_key_version: "encryptionkey@0"
I20260808 15:47:47.794843 15966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260808 15:47:47.798271 15982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:47.799729 15966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:47:47.799921 15966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "23a30485e9c341a7bdf5bdc9d6ec1158"
format_stamp: "Formatted at 2026-08-08 15:47:47 on dist-test-slave-gp6n"
server_key: "d01e95b011b0543d12ae7bb1e9d62c34"
server_key_iv: "0c492a9c634cb668b587aab0c5e974de"
server_key_version: "encryptionkey@0"
I20260808 15:47:47.800138 15966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:47.805886 15966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:47.810452 15966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:47.810768 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:47.829661 15966 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:39303
I20260808 15:47:47.829692 16039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:39303 every 8 connection(s)
I20260808 15:47:47.831110 15966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:47.831554 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 15966
I20260808 15:47:47.831725 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:47.832309 24072 external_mini_cluster.cc:1428] Setting key fa34bf9a3b9a7e173884519bc3fc061e
I20260808 15:47:47.835618 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:47.840232 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:47.834240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55817 (local address 127.23.130.62:39303)
0808 15:47:47.835139 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:47.835151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:47.835191 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:47.836915 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:47.836940 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:47.837264 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:47.837578 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:47.838700 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:47.838710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:47.838721 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:47.838735 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:47.838814 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:47.838837 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:47.839023 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:47.839121 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 15:47:47.839306 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":443,"threads_started":1}
I20260808 15:47:47.844121 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:47.847695 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158: Bootstrap starting.
I20260808 15:47:47.850843 16040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:47.851393 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:47.848589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41613 (local address 127.23.130.62:39303)
0808 15:47:47.848805 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:47.848814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:47.849391 (+ 577us) server_negotiation.cc:407] Connection header received
0808 15:47:47.849593 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:47.849604 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:47.849652 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:47.849751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:47.850712 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:47.850717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:47.850722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:47.850739 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:47.850791 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:47.850806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:47.850927 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:47.851048 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 15:47:47.851109 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:47:47.852115 16040 log.cc:824] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:47.854941 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158: No bootstrap required, opened a new log
I20260808 15:47:47.862177 16040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } }
I20260808 15:47:47.862637 16040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:47.862743 16040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a30485e9c341a7bdf5bdc9d6ec1158, State: Initialized, Role: FOLLOWER
I20260808 15:47:47.863298 16040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } }
I20260808 15:47:47.863534 16040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:47.863688 16040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:47.863857 16040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:47.866010 16040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } }
I20260808 15:47:47.866573 16040 leader_election.cc:302] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23a30485e9c341a7bdf5bdc9d6ec1158; no voters:
I20260808 15:47:47.867213 16040 leader_election.cc:288] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:47.867339 16050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:47.868052 16050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [term 1 LEADER]: Becoming Leader. State: Replica: 23a30485e9c341a7bdf5bdc9d6ec1158, State: Running, Role: LEADER
I20260808 15:47:47.868592 16050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } }
I20260808 15:47:47.869329 16040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:47.871557 16052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23a30485e9c341a7bdf5bdc9d6ec1158. Latest consensus state: current_term: 1 leader_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } } }
I20260808 15:47:47.872030 16052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:47.872584 16051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a30485e9c341a7bdf5bdc9d6ec1158" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 39303 } } }
I20260808 15:47:47.872920 16051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:47.873538 16058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:47.879006 16058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:47.890144 16058 catalog_manager.cc:1347] Generated new cluster ID: fa1e7a954a594a718bffc733e38297d7
I20260808 15:47:47.890270 16058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:47.902493 16058 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:47.902976 16058 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:47.914597 16058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23a30485e9c341a7bdf5bdc9d6ec1158: Generated new TSK 0
I20260808 15:47:47.915536 16058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:47.931696 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15966
W20260808 15:47:47.932857 16035 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55817
W20260808 15:47:47.933619 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 15966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:47.933809 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 15966
2026-08-08T15:47:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:48 dist-test-slave-gp6n krb5kdc[16075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:48 dist-test-slave-gp6n krb5kdc[16075](info): set up 2 sockets
Aug 08 15:47:48 dist-test-slave-gp6n krb5kdc[16075](info): commencing operation
krb5kdc: starting...
W20260808 15:47:49.149745 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:49 dist-test-slave-gp6n krb5kdc[16075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:49.479598 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34797
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:49.822000 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:49.822341 16092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:49.822532 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:49.829690 16092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:49.829843 16092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:49.829960 16092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:49.830081 16092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:49.830170 16092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:49.839915 16092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:49.841248 16092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:49.842648 16092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:49.854571 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:49.854663 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:49.854641 16101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:49.855851 16092 server_base.cc:1034] running on GCE node
I20260808 15:47:49.856592 16092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:49.858008 16092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:49.859203 16092 hybrid_clock.cc:648] HybridClock initialized: now 1786204069859165 us; error 32 us; skew 500 ppm
I20260808 15:47:49.859571 16092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:49 dist-test-slave-gp6n krb5kdc[16075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204069, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:49.867378 16092 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:49.869004 16092 webserver.cc:460] Webserver started at http://127.23.130.62:36215/ using document root <none> and password file <none>
I20260808 15:47:49.869709 16092 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:49.869859 16092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:49.870169 16092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:49.873250 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "cf5f2f108a034661baad50e486cf42b7"
format_stamp: "Formatted at 2026-08-08 15:47:49 on dist-test-slave-gp6n"
server_key: "d0b590e3de92b14d6e4a23b5e7a16d5c"
server_key_iv: "60fb22e92ceba19cb9232282dbab7dc0"
server_key_version: "encryptionkey@0"
I20260808 15:47:49.874061 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "cf5f2f108a034661baad50e486cf42b7"
format_stamp: "Formatted at 2026-08-08 15:47:49 on dist-test-slave-gp6n"
server_key: "d0b590e3de92b14d6e4a23b5e7a16d5c"
server_key_iv: "60fb22e92ceba19cb9232282dbab7dc0"
server_key_version: "encryptionkey@0"
I20260808 15:47:49.879407 16092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:47:49.882961 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:49.884281 16092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:47:49.884506 16092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "cf5f2f108a034661baad50e486cf42b7"
format_stamp: "Formatted at 2026-08-08 15:47:49 on dist-test-slave-gp6n"
server_key: "d0b590e3de92b14d6e4a23b5e7a16d5c"
server_key_iv: "60fb22e92ceba19cb9232282dbab7dc0"
server_key_version: "encryptionkey@0"
I20260808 15:47:49.884721 16092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:49.901592 16092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:49.905521 16092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:49.905810 16092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:49.924758 16092 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:34797
I20260808 15:47:49.924806 16169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:34797 every 8 connection(s)
I20260808 15:47:49.926090 16092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:49.930315 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:49.931517 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16092
I20260808 15:47:49.931718 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:49.932186 24072 external_mini_cluster.cc:1428] Setting key fa9fbac9f4b89b674460099fcd8b4776
I20260808 15:47:49.942945 16170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7: Bootstrap starting.
Aug 08 15:47:49 dist-test-slave-gp6n krb5kdc[16075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:49.946210 16170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:49.947366 16170 log.cc:824] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:49.950276 16170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7: No bootstrap required, opened a new log
I20260808 15:47:49.954077 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:49.934082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35263 (local address 127.23.130.62:34797)
0808 15:47:49.934889 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:49.934904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:49.934944 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:49.936597 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:49.936620 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:49.937017 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:49.937405 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:49.938943 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:49.940243 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:49.941480 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:49.942166 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:49.942358 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:49.946753 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:49.946761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:49.946775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:49.946810 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:49.949960 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:49.950772 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:49.950786 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:49.950792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:49.950967 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:49.951379 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:49.951383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:49.951385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:49.951901 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:49.952255 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:49.952692 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 15:47:49.953119 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260808 15:47:49.957821 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:49.958567 16170 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } }
I20260808 15:47:49.958926 16170 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:49.959031 16170 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5f2f108a034661baad50e486cf42b7, State: Initialized, Role: FOLLOWER
I20260808 15:47:49.959564 16170 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } }
I20260808 15:47:49.959815 16170 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:49.959971 16170 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:49.960139 16170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:49.963114 16170 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } }
I20260808 15:47:49.963613 16170 leader_election.cc:302] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5f2f108a034661baad50e486cf42b7; no voters:
I20260808 15:47:49.964290 16170 leader_election.cc:288] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:49.964577 16182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:49.966323 16182 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [term 1 LEADER]: Becoming Leader. State: Replica: cf5f2f108a034661baad50e486cf42b7, State: Running, Role: LEADER
I20260808 15:47:49.966876 16170 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:49.966954 16182 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } }
I20260808 15:47:49.978668 16183 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5f2f108a034661baad50e486cf42b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } } }
I20260808 15:47:49.979024 16183 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:49.978147 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5f2f108a034661baad50e486cf42b7. Latest consensus state: current_term: 1 leader_uuid: "cf5f2f108a034661baad50e486cf42b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f2f108a034661baad50e486cf42b7" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34797 } } }
I20260808 15:47:49.979573 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:49.983330 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:49.962554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59323 (local address 127.23.130.62:34797)
0808 15:47:49.962708 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:49.962713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:49.963169 (+ 456us) server_negotiation.cc:407] Connection header received
0808 15:47:49.963355 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:49.963360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:49.963432 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:49.963540 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:49.966077 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:49.967285 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:49.969213 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:49.970173 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:49.970339 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:49.974196 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:49.974207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:49.974215 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:49.974270 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:49.978519 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:49.979487 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:49.979492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:49.979496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:49.979646 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:49.980145 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:49.980152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:49.980157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:49.980672 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:49.982330 (+ 1658us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:49.982528 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 15:47:49.982803 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:47:49.987471 16197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:49.992086 16197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:50.000217 16197 catalog_manager.cc:1347] Generated new cluster ID: b0fd77b247a543e9985842ed468cacbd
I20260808 15:47:50.000378 16197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:50.009927 16197 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:50.011965 16197 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:50.026547 16197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf5f2f108a034661baad50e486cf42b7: Generated new TSK 0
I20260808 15:47:50.027421 16197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:50.038852 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16092
I20260808 15:47:50.039969 16164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:50.131341 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16092
2026-08-08T15:47:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:50 dist-test-slave-gp6n krb5kdc[16209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:50 dist-test-slave-gp6n krb5kdc[16209](info): set up 2 sockets
Aug 08 15:47:50 dist-test-slave-gp6n krb5kdc[16209](info): commencing operation
krb5kdc: starting...
W20260808 15:47:51.327790 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:51 dist-test-slave-gp6n krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:51.649648 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43469
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:51.987946 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:51.988312 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:51.988477 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:51.995576 16224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:51.995738 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:51.995854 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:51.995965 16224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:51.996052 16224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:52.006110 16224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:52.007535 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:52.008906 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:52.021027 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:52.021066 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:52.021121 16234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:52.022279 16224 server_base.cc:1034] running on GCE node
I20260808 15:47:52.023154 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:52.024583 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:52.025823 16224 hybrid_clock.cc:648] HybridClock initialized: now 1786204072025785 us; error 47 us; skew 500 ppm
I20260808 15:47:52.026185 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:52 dist-test-slave-gp6n krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204072, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:52.029647 16224 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:52.031247 16224 webserver.cc:460] Webserver started at http://127.23.130.62:34411/ using document root <none> and password file <none>
I20260808 15:47:52.031965 16224 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:52.032117 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:52.032440 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:52.035955 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "394dadb5dbdc4a36abbc9cea4410d3de"
format_stamp: "Formatted at 2026-08-08 15:47:52 on dist-test-slave-gp6n"
server_key: "b604b3af159dcca4556a189b6fb00501"
server_key_iv: "b1c26eb3f2ba9bd954e51f4aae415ae6"
server_key_version: "encryptionkey@0"
I20260808 15:47:52.036747 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "394dadb5dbdc4a36abbc9cea4410d3de"
format_stamp: "Formatted at 2026-08-08 15:47:52 on dist-test-slave-gp6n"
server_key: "b604b3af159dcca4556a189b6fb00501"
server_key_iv: "b1c26eb3f2ba9bd954e51f4aae415ae6"
server_key_version: "encryptionkey@0"
I20260808 15:47:52.042141 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:47:52.045660 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:52.047082 16224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:47:52.047269 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "394dadb5dbdc4a36abbc9cea4410d3de"
format_stamp: "Formatted at 2026-08-08 15:47:52 on dist-test-slave-gp6n"
server_key: "b604b3af159dcca4556a189b6fb00501"
server_key_iv: "b1c26eb3f2ba9bd954e51f4aae415ae6"
server_key_version: "encryptionkey@0"
I20260808 15:47:52.047477 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:52.065914 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:52.070011 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:52.070307 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:52.089552 16224 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:43469
I20260808 15:47:52.089601 16296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:43469 every 8 connection(s)
I20260808 15:47:52.090997 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:52.094785 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:52.100445 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16224
I20260808 15:47:52.100643 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:52.101240 24072 external_mini_cluster.cc:1428] Setting key 9c2e99853fb7e68e7f4032b1459a2f2b
I20260808 15:47:52.106163 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de: Bootstrap starting.
I20260808 15:47:52.109784 16297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:52.111095 16297 log.cc:824] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:52.114646 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de: No bootstrap required, opened a new log
I20260808 15:47:52.116314 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:52.103336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54809 (local address 127.23.130.62:43469)
0808 15:47:52.104239 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:52.104255 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:52.104303 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:47:52.106091 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:52.106114 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:52.106639 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:52.106994 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:52.107602 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:52.109019 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:52.111005 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:52.111821 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:52.112060 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:52.113432 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:52.113448 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:52.113465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:52.114184 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:52.114280 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:52.114313 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:52.114458 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:52.114749 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:47:52.115012 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":430,"threads_started":1}
I20260808 15:47:52.120584 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:52.125284 16297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } }
I20260808 15:47:52.125789 16297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:52.125927 16297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394dadb5dbdc4a36abbc9cea4410d3de, State: Initialized, Role: FOLLOWER
I20260808 15:47:52.126700 16297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } }
I20260808 15:47:52.127017 16297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:52.127172 16297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:52.127364 16297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:52.130241 16297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } }
I20260808 15:47:52.130868 16297 leader_election.cc:302] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394dadb5dbdc4a36abbc9cea4410d3de; no voters:
I20260808 15:47:52.131611 16297 leader_election.cc:288] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:52.131733 16308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:52.133311 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:52.125924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34877 (local address 127.23.130.62:43469)
0808 15:47:52.126131 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:52.126137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:52.126703 (+ 566us) server_negotiation.cc:407] Connection header received
0808 15:47:52.126809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:52.126814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:52.126901 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:52.127018 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:52.127662 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:52.128792 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:52.130121 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:52.130863 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:52.130998 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:52.132353 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:52.132363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:52.132371 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:52.132401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:52.132474 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:52.132502 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:52.132581 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:52.132883 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 15:47:52.132978 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 15:47:52.133566 16308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [term 1 LEADER]: Becoming Leader. State: Replica: 394dadb5dbdc4a36abbc9cea4410d3de, State: Running, Role: LEADER
I20260808 15:47:52.134068 16297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:52.134244 16308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } }
I20260808 15:47:52.137247 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394dadb5dbdc4a36abbc9cea4410d3de. Latest consensus state: current_term: 1 leader_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } } }
I20260808 15:47:52.137442 16309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394dadb5dbdc4a36abbc9cea4410d3de" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 43469 } } }
I20260808 15:47:52.137701 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:52.137704 16309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:52.140540 16319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:52.144716 16319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:52.153915 16319 catalog_manager.cc:1347] Generated new cluster ID: 38a6307c7d9d4739a7cd5943ed7cd6b3
I20260808 15:47:52.154045 16319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:52.183529 16319 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:52.184895 16319 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:52.196929 16319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394dadb5dbdc4a36abbc9cea4410d3de: Generated new TSK 0
I20260808 15:47:52.197762 16319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:52.203728 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16224
W20260808 15:47:52.204870 16292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54809
W20260808 15:47:52.205587 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16224: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:52.205745 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16224
2026-08-08T15:47:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:52 dist-test-slave-gp6n krb5kdc[16331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:52 dist-test-slave-gp6n krb5kdc[16331](info): set up 2 sockets
Aug 08 15:47:52 dist-test-slave-gp6n krb5kdc[16331](info): commencing operation
krb5kdc: starting...
W20260808 15:47:53.501276 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:53 dist-test-slave-gp6n krb5kdc[16331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:53.815541 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37017
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:54.161278 16345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:54.161576 16345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:54.161760 16345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:54.169189 16345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:54.169353 16345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:54.169445 16345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:54.169536 16345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:54.169621 16345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:54.179939 16345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:54.181247 16345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:54.182717 16345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:54.194845 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:54.194877 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:54.194944 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:54.199942 16345 server_base.cc:1034] running on GCE node
I20260808 15:47:54.200723 16345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:54.202173 16345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:54.203397 16345 hybrid_clock.cc:648] HybridClock initialized: now 1786204074203325 us; error 60 us; skew 500 ppm
I20260808 15:47:54.203776 16345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:54 dist-test-slave-gp6n krb5kdc[16331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:54.207237 16345 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:54.208843 16345 webserver.cc:460] Webserver started at http://127.23.130.62:40001/ using document root <none> and password file <none>
I20260808 15:47:54.209533 16345 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:54.209638 16345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:54.209909 16345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:54.212972 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "850a48261c834c91bb023b7b0c35b75d"
format_stamp: "Formatted at 2026-08-08 15:47:54 on dist-test-slave-gp6n"
server_key: "da759d4b8827268fca3d24c2b0f10569"
server_key_iv: "e8118a76c401b205de4a32374adeb91f"
server_key_version: "encryptionkey@0"
I20260808 15:47:54.213686 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "850a48261c834c91bb023b7b0c35b75d"
format_stamp: "Formatted at 2026-08-08 15:47:54 on dist-test-slave-gp6n"
server_key: "da759d4b8827268fca3d24c2b0f10569"
server_key_iv: "e8118a76c401b205de4a32374adeb91f"
server_key_version: "encryptionkey@0"
I20260808 15:47:54.220479 16345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260808 15:47:54.225498 16362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:54.227069 16345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:47:54.227211 16345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "850a48261c834c91bb023b7b0c35b75d"
format_stamp: "Formatted at 2026-08-08 15:47:54 on dist-test-slave-gp6n"
server_key: "da759d4b8827268fca3d24c2b0f10569"
server_key_iv: "e8118a76c401b205de4a32374adeb91f"
server_key_version: "encryptionkey@0"
I20260808 15:47:54.227384 16345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:54.233048 16345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:54.237059 16345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:54.237303 16345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:54.255443 16345 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37017
I20260808 15:47:54.255486 16418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37017 every 8 connection(s)
I20260808 15:47:54.256891 16345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:54.257434 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16345
I20260808 15:47:54.257642 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:54.258278 24072 external_mini_cluster.cc:1428] Setting key f05fb761a20d0ca5e0170ee89adb2f43
I20260808 15:47:54.261323 16419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:54.266656 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:54.260211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38679 (local address 127.23.130.62:37017)
0808 15:47:54.261037 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:54.261049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:54.261091 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:47:54.263024 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:54.263047 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:54.263368 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:54.263817 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:54.265001 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:54.265007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:54.265020 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:54.265045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:54.265117 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:54.265137 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:54.265350 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:54.265451 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 15:47:54.265668 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":368,"threads_started":1}
I20260808 15:47:54.270242 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:54.273304 16419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d: Bootstrap starting.
I20260808 15:47:54.276494 16419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:54.277522 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:54.274733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35601 (local address 127.23.130.62:37017)
0808 15:47:54.274977 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:54.274986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:54.275492 (+ 506us) server_negotiation.cc:407] Connection header received
0808 15:47:54.275678 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:54.275684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:54.275745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:54.275846 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:54.276834 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:54.276841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:54.276845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:47:54.276873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:54.276914 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:47:54.276929 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:54.277046 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:54.277158 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 15:47:54.277216 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:47:54.277681 16419 log.cc:824] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:54.280426 16419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d: No bootstrap required, opened a new log
I20260808 15:47:54.287591 16419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } }
I20260808 15:47:54.287957 16419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:47:54.288064 16419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850a48261c834c91bb023b7b0c35b75d, State: Initialized, Role: FOLLOWER
I20260808 15:47:54.288605 16419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } }
I20260808 15:47:54.288843 16419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:47:54.288987 16419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:47:54.289155 16419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:54.291604 16419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } }
I20260808 15:47:54.292073 16419 leader_election.cc:302] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850a48261c834c91bb023b7b0c35b75d; no voters:
I20260808 15:47:54.292704 16419 leader_election.cc:288] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:47:54.292811 16429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:54.293541 16429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [term 1 LEADER]: Becoming Leader. State: Replica: 850a48261c834c91bb023b7b0c35b75d, State: Running, Role: LEADER
I20260808 15:47:54.294078 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } }
I20260808 15:47:54.294831 16419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:54.296939 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850a48261c834c91bb023b7b0c35b75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } } }
I20260808 15:47:54.297430 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:54.299501 16431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850a48261c834c91bb023b7b0c35b75d. Latest consensus state: current_term: 1 leader_uuid: "850a48261c834c91bb023b7b0c35b75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a48261c834c91bb023b7b0c35b75d" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37017 } } }
I20260808 15:47:54.300045 16435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:54.300185 16431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:54.305037 16435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:54.313839 16435 catalog_manager.cc:1347] Generated new cluster ID: 974b261bb71348d0ac239b43f15fbc77
I20260808 15:47:54.313951 16435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:54.332101 16435 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:54.332600 16435 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:54.341261 16435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 850a48261c834c91bb023b7b0c35b75d: Generated new TSK 0
I20260808 15:47:54.342069 16435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:54.352442 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16345
W20260808 15:47:54.353655 16414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38679
W20260808 15:47:54.354451 24072 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16345: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:47:54.354620 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16345
2026-08-08T15:47:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2143 ms)
[----------] 18 tests from AuthTokenIssuingTest (38304 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:54 dist-test-slave-gp6n krb5kdc[16451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:54 dist-test-slave-gp6n krb5kdc[16451](info): set up 2 sockets
Aug 08 15:47:54 dist-test-slave-gp6n krb5kdc[16451](info): commencing operation
krb5kdc: starting...
W20260808 15:47:55.565218 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.011s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:47:55 dist-test-slave-gp6n krb5kdc[16451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:47:55.894110 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34205
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:47:56.242357 16466 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:47:56.242782 16466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:56.242902 16466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:56.243054 16466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:56.250734 16466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:56.250839 16466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:56.250943 16466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:56.251047 16466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:56.251166 16466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:56.261096 16466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:34205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:56.262475 16466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:56.263867 16466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:56.276013 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:56.276103 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:56.276124 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:56.276919 16466 server_base.cc:1034] running on GCE node
I20260808 15:47:56.277630 16466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:56.279071 16466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:56.280292 16466 hybrid_clock.cc:648] HybridClock initialized: now 1786204076280214 us; error 54 us; skew 500 ppm
I20260808 15:47:56.280740 16466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:56 dist-test-slave-gp6n krb5kdc[16451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:56.288699 16466 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:47:56.290264 16466 webserver.cc:460] Webserver started at http://127.23.130.62:35519/ using document root <none> and password file <none>
I20260808 15:47:56.291075 16466 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:56.291200 16466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:56.291556 16466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:56.294761 16466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "83d1854335074e9694ca52f554f50aa9"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "315c1c5314add0d5db60a5a143991781"
server_key_iv: "2d59331e1e62be5c3190e6da4f79d258"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.295608 16466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "83d1854335074e9694ca52f554f50aa9"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "315c1c5314add0d5db60a5a143991781"
server_key_iv: "2d59331e1e62be5c3190e6da4f79d258"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.300899 16466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:56.304374 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:56.305892 16466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:47:56.306083 16466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "83d1854335074e9694ca52f554f50aa9"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "315c1c5314add0d5db60a5a143991781"
server_key_iv: "2d59331e1e62be5c3190e6da4f79d258"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.306315 16466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:56.324085 16466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:56.328683 16466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:56.328970 16466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:56.348353 16466 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:34205
I20260808 15:47:56.348507 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:34205 every 8 connection(s)
I20260808 15:47:56.349756 16466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:47:56.353583 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:56.356319 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16466
I20260808 15:47:56.356596 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:47:56.357277 24072 external_mini_cluster.cc:1428] Setting key 1b7636793e87fafff14a8f8b69b33dab
I20260808 15:47:56.360889 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:47:56 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:56.387527 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.364159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34205 (local address 127.0.0.1:58896)
0808 15:47:56.365087 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.365139 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:56.368112 (+ 2973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:56.369123 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:56.369134 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:56.370053 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:56.370624 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.370652 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.372471 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.372475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.373570 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.373582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.375123 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.375129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.375260 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.376011 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:56.376033 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:56.378516 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:56.383212 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.383227 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.383240 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.383798 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.384332 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.384335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.384337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.384526 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.385590 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:56.385596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:56.385787 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 15:47:56.386004 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 15:47:56.386382 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":368,"threads_started":1}
I20260808 15:47:56.387629 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.364159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.23.130.62:34205)
0808 15:47:56.366233 (+ 2074us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:56.366242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:56.366304 (+ 62us) server_negotiation.cc:407] Connection header received
0808 15:47:56.368567 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:56.368573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:56.368799 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:56.368959 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:56.370853 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.372280 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.373711 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.374858 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.375085 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.378756 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:56.378769 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:56.378788 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:56.378834 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:56.383055 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.383979 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.383990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.383996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.384201 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.384665 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.384672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.384678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.385439 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:56.385652 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:56.386241 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 15:47:56.386598 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"spinlock_wait_cycles":25344,"thread_start_us":1452,"threads_started":1}
I20260808 15:47:56.391077 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:47:56.392359 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.391936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39386)
0808 15:47:56.392116 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.392243 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 15:47:56.392643 16486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.130.61@KRBTEST.COM; defaulting to no policy
I20260808 15:47:56.394330 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.393974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39394)
0808 15:47:56.394125 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.394192 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 15:47:56.394560 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:36293: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
I20260808 15:47:56.438376 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:36293
--webserver_interface=127.23.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:47:56.443869 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } attempt: 1
I20260808 15:47:56.445204 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.444593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39400)
0808 15:47:56.444860 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.445009 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:47:56.446337 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.445895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39408)
0808 15:47:56.446068 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.446163 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:47:56.446591 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:36293: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
I20260808 15:47:56.514885 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } attempt: 2
I20260808 15:47:56.516188 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.515676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39412)
0808 15:47:56.515932 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.516042 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 15:47:56.517292 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.516923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39422)
0808 15:47:56.517086 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.517165 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 15:47:56.517539 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:36293: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
I20260808 15:47:56.608804 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } attempt: 3
I20260808 15:47:56.610083 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.609643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39438)
0808 15:47:56.609841 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.609933 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:47:56.611336 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.610798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39442)
0808 15:47:56.611010 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.611121 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 15:47:56.611577 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:36293: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
I20260808 15:47:56.754863 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } attempt: 4
I20260808 15:47:56.756176 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.755621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39446)
0808 15:47:56.755948 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.756039 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260808 15:47:56.757232 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.756882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39450)
0808 15:47:56.757026 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.757112 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 15:47:56.757477 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:36293: Network error: Client connection negotiation failed: client connection to 127.23.130.61:36293: connect: Connection refused (error 111)
W20260808 15:47:56.783432 16549 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:47:56.783762 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:56.783828 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:56.783973 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:56.791304 16549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:56.791394 16549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:56.791496 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:56.791585 16549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:56.791695 16549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:56.801571 16549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:56.802857 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:56.804330 16549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:56.816572 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:56.816622 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:56.816653 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:56.817493 16549 server_base.cc:1034] running on GCE node
I20260808 15:47:56.818178 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:56.819538 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:56.820767 16549 hybrid_clock.cc:648] HybridClock initialized: now 1786204076820709 us; error 29 us; skew 500 ppm
I20260808 15:47:56.821134 16549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:56 dist-test-slave-gp6n krb5kdc[16451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:56.824484 16549 init.cc:377] Logged in from keytab as kudu/127.23.130.61@KRBTEST.COM (short username kudu)
I20260808 15:47:56.825868 16549 webserver.cc:460] Webserver started at http://127.23.130.61:38037/ using document root <none> and password file <none>
I20260808 15:47:56.826615 16549 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:56.826735 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:56.827085 16549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:56.831130 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data/instance:
uuid: "79c4cb807a824e42a626c4e0752f86d8"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "7d0257ce6168e1ab188ce79a30d7305f"
server_key_iv: "c1d0b004d4e01126f0981a9fac7b73e1"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.832296 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal/instance:
uuid: "79c4cb807a824e42a626c4e0752f86d8"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "7d0257ce6168e1ab188ce79a30d7305f"
server_key_iv: "c1d0b004d4e01126f0981a9fac7b73e1"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.839754 16549 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260808 15:47:56.844938 16565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:56.846673 16549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:47:56.846922 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "79c4cb807a824e42a626c4e0752f86d8"
format_stamp: "Formatted at 2026-08-08 15:47:56 on dist-test-slave-gp6n"
server_key: "7d0257ce6168e1ab188ce79a30d7305f"
server_key_iv: "c1d0b004d4e01126f0981a9fac7b73e1"
server_key_version: "encryptionkey@0"
I20260808 15:47:56.847218 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:56.875648 16549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:56.880734 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:56.881067 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:56.901742 16549 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.61:36293
I20260808 15:47:56.901746 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.61:36293 every 8 connection(s)
I20260808 15:47:56.903182 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
I20260808 15:47:56.906989 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:56.912732 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16549
I20260808 15:47:56.912935 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-1/wal/instance
I20260808 15:47:56.913373 24072 external_mini_cluster.cc:1428] Setting key 57287de44b42cb8132a6cdb01afd1a75
I20260808 15:47:56.913795 16619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:47:56 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:56.935895 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.916395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.23.130.62:34205)
0808 15:47:56.916947 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:56.916951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:56.917952 (+ 1001us) server_negotiation.cc:407] Connection header received
0808 15:47:56.919884 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:56.919890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:56.919983 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:56.920124 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:56.921673 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.922537 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.924063 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.924485 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.924630 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.928430 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:56.928435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:56.928438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:56.928501 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:56.932474 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.933224 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.933232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.933239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.933398 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.933815 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.933823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.933829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.934244 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:56.934432 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:56.935139 (+ 707us) server_negotiation.cc:299] Negotiation successful
0808 15:47:56.935395 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":323,"threads_started":1}
I20260808 15:47:56.936544 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.917048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34205 (local address 127.0.0.1:58898)
0808 15:47:56.917791 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.917830 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:56.919617 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:56.920366 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:56.920374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:56.920888 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:56.921444 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.921460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.922708 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.922712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.923900 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.923912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.924724 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.924732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.924966 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.925692 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:56.925715 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:56.928182 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:56.932638 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.932645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.932658 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.933089 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.933527 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.933530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.933532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.933703 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.934365 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:56.934370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:56.934767 (+ 397us) client_negotiation.cc:769] Sending connection context
0808 15:47:56.935009 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 15:47:56.935383 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":307,"threads_started":1}
I20260808 15:47:56.938436 16619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:47:56 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.60@KRBTEST.COM" created.
I20260808 15:47:56.954387 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.939325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39466)
0808 15:47:56.939516 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.939537 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:56.939639 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:56.940639 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:56.940643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:56.941076 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:56.941200 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.941208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.942677 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.942685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.943514 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:56.943525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.944463 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.944468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:56.944642 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.945225 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:56.945242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:56.947715 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:56.951738 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.951742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.951744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.952132 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.952582 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:56.952585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:56.952587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:56.952736 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:56.953530 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:56.953536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:56.953683 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 15:47:56.953831 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 15:47:56.954041 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:47:56.954819 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.939585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.23.130.61:36293)
0808 15:47:56.940241 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:56.940246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:56.940270 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:47:56.940311 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:56.940317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:56.940483 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:56.940647 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:56.941377 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.942550 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.943650 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:56.944302 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:56.944448 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:56.947918 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:56.947929 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:56.947941 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:56.947974 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:56.951582 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.952290 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.952299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.952303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.952460 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:56.952861 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:56.952869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:56.952875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:56.953360 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:56.953602 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:56.953997 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 15:47:56.954281 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":383,"threads_started":1}
I20260808 15:47:56.957217 16619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:47:56.958299 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.957850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36696)
0808 15:47:56.958044 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.958152 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:47:56.958576 16568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:56.960233 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:56.959892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36708)
0808 15:47:56.960042 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:56.960133 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:47:56.960489 16619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
I20260808 15:47:56.989550 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:45263
--webserver_interface=127.23.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:47:57.009826 16619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 1
I20260808 15:47:57.011380 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.010667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36720)
0808 15:47:57.011012 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.011142 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260808 15:47:57.012583 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.012098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36722)
0808 15:47:57.012305 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.012415 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:47:57.012877 16619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.056733 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } attempt: 5
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:47:57.071673 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.057500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39482)
0808 15:47:57.057700 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.057722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.057836 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.058237 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.058240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.058654 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.058835 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.058843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.060231 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.060234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.061082 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.061092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.062060 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.062064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.062228 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.062910 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.062927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.065249 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.069241 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.069245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.069248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.069624 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.070008 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.070012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.070016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.070140 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.070754 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.070757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.070888 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.071003 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.071341 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 15:47:57.071679 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.057577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.23.130.61:36293)
0808 15:47:57.057765 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.057771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.057797 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:47:57.057965 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.057971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.058075 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.058201 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.058992 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.060097 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.061239 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.061899 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.062021 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.065442 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.065449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.065455 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.065502 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.069110 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.069740 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.069746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.069749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.069899 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.070244 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.070248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.070251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.070642 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.070792 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.071131 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.071373 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:47:57.072845 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:47:57.073854 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.073482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36728)
0808 15:47:57.073647 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.073721 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:47:57.074815 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.074527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36742)
0808 15:47:57.074646 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.074724 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 15:47:57.075099 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.081171 16619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 2
I20260808 15:47:57.082124 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.081686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36748)
0808 15:47:57.081868 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.081962 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:47:57.083101 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.082736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36758)
0808 15:47:57.082881 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.082963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:47:57.083320 16619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.126359 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 1
I20260808 15:47:57.127643 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.127172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36762)
0808 15:47:57.127369 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.127463 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 15:47:57.128602 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.128313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36774)
0808 15:47:57.128437 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.128500 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:47:57.128840 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.174578 16619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 3
I20260808 15:47:57.176028 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.175396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36786)
0808 15:47:57.175661 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.175793 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 15:47:57.177338 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.176769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36800)
0808 15:47:57.177037 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.177150 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260808 15:47:57.177627 16619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.197158 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 2
I20260808 15:47:57.198534 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.197981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36808)
0808 15:47:57.198209 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.198297 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:47:57.199577 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.199262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36824)
0808 15:47:57.199402 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.199471 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:47:57.199828 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.306074 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 3
I20260808 15:47:57.307497 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.306946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36836)
0808 15:47:57.307197 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.307291 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 15:47:57.308725 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.308336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36842)
0808 15:47:57.308528 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.308617 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 15:47:57.309070 16542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
I20260808 15:47:57.320881 16619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 4
I20260808 15:47:57.322088 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.321546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36854)
0808 15:47:57.321774 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.321889 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 15:47:57.323216 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.322840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36856)
0808 15:47:57.323004 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.323088 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:47:57.323443 16619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:45263: Network error: Client connection negotiation failed: client connection to 127.23.130.60:45263: connect: Connection refused (error 111)
W20260808 15:47:57.339396 16627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:47:57.339769 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:47:57.339890 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:47:57.340021 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:47:57.347355 16627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:47:57.347492 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:47:57.347592 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:47:57.347666 16627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:47:57.347756 16627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:47:57.361673 16627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:34205,127.23.130.61:36293,127.23.130.60:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:47:57.362954 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:47:57.364221 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:47:57.376456 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:57.376417 16636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:47:57.376377 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:47:57.377267 16627 server_base.cc:1034] running on GCE node
I20260808 15:47:57.377983 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:47:57.379426 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:47:57.380712 16627 hybrid_clock.cc:648] HybridClock initialized: now 1786204077380643 us; error 32 us; skew 500 ppm
I20260808 15:47:57.381090 16627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204077, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:47:57.384457 16627 init.cc:377] Logged in from keytab as kudu/127.23.130.60@KRBTEST.COM (short username kudu)
I20260808 15:47:57.386003 16627 webserver.cc:460] Webserver started at http://127.23.130.60:40653/ using document root <none> and password file <none>
I20260808 15:47:57.386785 16627 fs_manager.cc:359] Metadata directory not provided
I20260808 15:47:57.386920 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:47:57.387250 16627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:47:57.390204 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data/instance:
uuid: "62520b8b0ba94fe6bd086a9a4c8a493f"
format_stamp: "Formatted at 2026-08-08 15:47:57 on dist-test-slave-gp6n"
server_key: "d743155e73f47554695c15801747dbd3"
server_key_iv: "430845fc0f545171ab59cb8c5b30b542"
server_key_version: "encryptionkey@0"
I20260808 15:47:57.391117 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal/instance:
uuid: "62520b8b0ba94fe6bd086a9a4c8a493f"
format_stamp: "Formatted at 2026-08-08 15:47:57 on dist-test-slave-gp6n"
server_key: "d743155e73f47554695c15801747dbd3"
server_key_iv: "430845fc0f545171ab59cb8c5b30b542"
server_key_version: "encryptionkey@0"
I20260808 15:47:57.396351 16627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:47:57.399796 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:47:57.401177 16627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:47:57.401360 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "62520b8b0ba94fe6bd086a9a4c8a493f"
format_stamp: "Formatted at 2026-08-08 15:47:57 on dist-test-slave-gp6n"
server_key: "d743155e73f47554695c15801747dbd3"
server_key_iv: "430845fc0f545171ab59cb8c5b30b542"
server_key_version: "encryptionkey@0"
I20260808 15:47:57.401587 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:47:57.426568 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:47:57.431058 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:47:57.431365 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:47:57.451607 16627 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.60:45263
I20260808 15:47:57.451699 16699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.60:45263 every 8 connection(s)
I20260808 15:47:57.453116 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
I20260808 15:47:57.456995 16700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:47:57.461994 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16627
I20260808 15:47:57.462190 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786203706771851-24072-0/minicluster-data/master-2/wal/instance
I20260808 15:47:57.462706 24072 external_mini_cluster.cc:1428] Setting key fd693f7459de5f7e43763faa3d6df1f9
I20260808 15:47:57.463740 16700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204077, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
I20260808 15:47:57.484068 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.466577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.23.130.62:34205)
0808 15:47:57.467058 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.467067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.467723 (+ 656us) server_negotiation.cc:407] Connection header received
0808 15:47:57.468382 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.468392 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.468476 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.468604 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.469966 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.470780 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.472643 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.473073 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.473168 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.477058 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.477067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.477073 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.477137 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.480300 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.481171 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.481176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.481179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.481336 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.481833 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.481838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.481841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.482195 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.482315 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.483263 (+ 948us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.483497 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
I20260808 15:47:57.485118 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.464668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.23.130.60:45263)
0808 15:47:57.465500 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.465511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.465551 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:47:57.468107 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.468127 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.468353 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.468515 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.469986 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.471405 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.472680 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.473273 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.473438 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.477825 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.477832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.477842 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.477869 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.480740 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.481484 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.481491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.481497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.481712 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.482122 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.482125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.482127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.482823 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.483160 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.483424 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.483766 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":218,"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":19328,"thread_start_us":380,"threads_started":1}
I20260808 15:47:57.485162 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.466673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:34205 (local address 127.0.0.1:58908)
0808 15:47:57.467560 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.467599 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.468120 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.468706 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.468713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.469424 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.469779 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.469796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.470935 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.470941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.472474 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.472486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.473301 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.473312 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.473612 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.474374 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.474444 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.476849 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.480444 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.480455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.480467 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.481053 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.481480 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.481484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.481487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.481712 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.482313 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.482321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.482825 (+ 504us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.483141 (+ 316us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.483668 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":149,"spinlock_wait_cycles":14080,"thread_start_us":370,"threads_started":1}
I20260808 15:47:57.486291 16542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 4
I20260808 15:47:57.486965 16700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204077, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:47:57.502841 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.486936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36874)
0808 15:47:57.487192 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.487212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.487312 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.488534 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.488541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.489031 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.489241 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.489253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.490835 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.490841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.491946 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.491964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.492943 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.492949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.493157 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.493682 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.493704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.496369 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.499917 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.499923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.499928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.500502 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.500950 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.500956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.500959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.501160 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.501693 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.501699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.501887 (+ 188us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.502085 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.502377 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260808 15:47:57.502918 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.487144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.23.130.60:45263)
0808 15:47:57.487536 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.487544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.487564 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:47:57.488292 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.488296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.488367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.489408 (+ 1041us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.489569 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.490718 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.492125 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.492762 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.493459 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.496578 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.496584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.496589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.496623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.499774 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.500652 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.500657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.500661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.500815 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.501272 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.501275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.501277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.501550 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.501728 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.502198 (+ 470us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.502492 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":1920}
I20260808 15:47:57.503583 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.487733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:36293 (local address 127.0.0.1:39490)
0808 15:47:57.487894 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.487915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.488030 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.488806 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.488812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.489293 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.489466 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.489474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.490529 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.490535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.491876 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.491889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.492632 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.492638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.492848 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.493476 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.493504 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.496153 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.500255 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.500263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.500269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.500823 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.501628 (+ 805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.501634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.501640 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.501845 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.502606 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.502611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.502763 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.502961 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.503260 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"spinlock_wait_cycles":6016}
I20260808 15:47:57.504339 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.488136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.23.130.61:36293)
0808 15:47:57.488324 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.488332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.488351 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:47:57.488411 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.488416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.488490 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.488613 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.489620 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.490336 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.492058 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.492474 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.492628 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.496348 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.496353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.496356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.496388 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.500124 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.501317 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.501323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.501326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.501494 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.501987 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.501991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.501994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.502354 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.502539 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.503893 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.504076 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:47:57.505554 16700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:47:57.507393 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.488744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.23.130.62:34205)
0808 15:47:57.488904 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.488910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.488925 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:47:57.488983 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.488988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.489093 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.489214 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.490764 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.491860 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.493682 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.494387 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.494545 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.499801 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.499810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.499817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.499860 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.503656 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.504530 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.504534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.504537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.504687 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.505153 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.505159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.505162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.505479 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.506780 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.506906 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.507087 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:47:57.510007 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: Bootstrap starting.
I20260808 15:47:57.514089 16542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:57.515583 16542 log.cc:824] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204077, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:47:57.518877 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: No bootstrap required, opened a new log
I20260808 15:47:57.522641 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.506553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.23.130.60:45263)
0808 15:47:57.506999 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.507004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.507017 (+ 13us) server_negotiation.cc:407] Connection header received
0808 15:47:57.507056 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.507062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.507123 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.507219 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.508128 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.509110 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.510786 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.511566 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.511685 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.516814 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.516820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.516825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.516863 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.519865 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.520674 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.520679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.520683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.520831 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.521256 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.521261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.521265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.521581 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.521741 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.522132 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.522310 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260808 15:47:57.523308 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.506273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36882)
0808 15:47:57.506697 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.506718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.506809 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.507281 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.507287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.507787 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.507977 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.507997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.509284 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.509290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.510630 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.510646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.512997 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.513006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.513221 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.513979 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.514004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.516609 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.519990 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.519997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.520001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.520522 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.520956 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.520963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.520968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.521138 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.521698 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.521703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.521847 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.522826 (+ 979us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.523092 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323}
I20260808 15:47:57.526986 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.509407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.23.130.61:36293)
0808 15:47:57.509780 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.509784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.509796 (+ 12us) server_negotiation.cc:407] Connection header received
0808 15:47:57.509833 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.509836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.509886 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.510043 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.511704 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.512785 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.514216 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.514714 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.514854 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.520202 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.520208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.520213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.520251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.523746 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.524592 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.524597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.524601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.524760 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.525225 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.525230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.525233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.525573 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.525802 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.526396 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.526651 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260808 15:47:57.527477 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: Bootstrap starting.
I20260808 15:47:57.529837 16542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.530256 16542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d1854335074e9694ca52f554f50aa9, State: Initialized, Role: FOLLOWER
I20260808 15:47:57.530824 16542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.531015 16700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:57.531840 16707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.532363 16707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.532406 16700 log.cc:824] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:57.532827 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:47:57.533164 16542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:57.535477 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: No bootstrap required, opened a new log
I20260808 15:47:57.536580 16513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "83d1854335074e9694ca52f554f50aa9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58914
I20260808 15:47:57.537184 16513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 15:47:57.537451 16513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:57.540138 16513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.543530 16513 leader_election.cc:288] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 79c4cb807a824e42a626c4e0752f86d8 (127.23.130.61:36293), 62520b8b0ba94fe6bd086a9a4c8a493f (127.23.130.60:45263)
I20260808 15:47:57.544453 16700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.544955 16700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62520b8b0ba94fe6bd086a9a4c8a493f, State: Initialized, Role: FOLLOWER
I20260808 15:47:57.545549 16700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.552583 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.552964 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.554693 16700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:47:57.554965 16720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 15:47:57.554800 16673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83d1854335074e9694ca52f554f50aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f"
W20260808 15:47:57.555066 16720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:47:57.555317 16673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:57.546170 16466 master_runner.cc:186] Error getting master registration for 127.23.130.61:36293: OK, instance_id { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" instance_seqno: 1786204076896880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:47:57.559412 16466 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:47:57.559227 16673 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d1854335074e9694ca52f554f50aa9 in term 1.
I20260808 15:47:57.560371 16486 leader_election.cc:302] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62520b8b0ba94fe6bd086a9a4c8a493f, 83d1854335074e9694ca52f554f50aa9; no voters:
I20260808 15:47:57.560917 16707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:47:57.581482 16707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 83d1854335074e9694ca52f554f50aa9, State: Running, Role: LEADER
W20260808 15:47:57.582152 16490 tablet.cc:2369] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:47:57.583472 16707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.591322 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d1854335074e9694ca52f554f50aa9. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.591841 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:57.593786 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.570898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.23.130.62:34205)
0808 15:47:57.571130 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.571139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.571618 (+ 479us) server_negotiation.cc:407] Connection header received
0808 15:47:57.571935 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.571941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.572036 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.572115 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.574965 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.576509 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.578980 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.579892 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.580026 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.584965 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.584972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.584977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.585026 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.589924 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.590930 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.590938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.590943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.591115 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.591649 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.591654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.591658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.592214 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.592476 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.593044 (+ 568us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.593321 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:47:57.596499 16741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:47:57.598748 16741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:47:57.615428 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.571992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.23.130.61:36293)
0808 15:47:57.572167 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.572174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.584095 (+ 11921us) server_negotiation.cc:407] Connection header received
0808 15:47:57.584453 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.584461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.584559 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.584637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.591801 (+ 7164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.592928 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.600467 (+ 7539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.601159 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.601257 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.606234 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.606242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.606248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.606299 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.610224 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.612377 (+ 2153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.612384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.612388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.612556 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.613137 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.613142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.613146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.613561 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.613807 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.614764 (+ 957us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.615031 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:47:57.618878 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.583210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.23.130.60:45263)
0808 15:47:57.583439 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.583448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.596989 (+ 13541us) server_negotiation.cc:407] Connection header received
0808 15:47:57.597295 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.597303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.597426 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.597506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.599690 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.600792 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.603564 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.604333 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.604451 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.610118 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.610125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.610130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.610175 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.613968 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.615362 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.615383 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.615389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.615562 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.616654 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.616672 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.616674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.617044 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.617312 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.618117 (+ 805us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.618532 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:47:57.621601 16673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [term 1 FOLLOWER]: Refusing update from remote peer 83d1854335074e9694ca52f554f50aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:57.622267 16746 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:47:57.622638 16619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } attempt: 5
I20260808 15:47:57.622612 16726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 08 15:47:57 dist-test-slave-gp6n krb5kdc[16451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
I20260808 15:47:57.643931 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d1854335074e9694ca52f554f50aa9. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.644268 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.647511 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.623504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:45263 (local address 127.0.0.1:36914)
0808 15:47:57.623979 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 15:47:57.624000 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:47:57.624095 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:47:57.625814 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:47:57.625820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:47:57.626344 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:47:57.626631 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.626647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.630152 (+ 3505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.630158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.631664 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:47:57.631682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.633284 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.633292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:47:57.633684 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.634636 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:47:57.634670 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:47:57.639438 (+ 4768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:47:57.644144 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.644152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.644158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.644839 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.645403 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:47:57.645407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:47:57.645410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:47:57.645589 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:47:57.646462 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:47:57.646467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:47:57.646624 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 15:47:57.646782 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 15:47:57.647097 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260808 15:47:57.629724 16627 master_runner.cc:186] Error getting master registration for 127.23.130.61:36293: OK, instance_id { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" instance_seqno: 1786204076896880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:47:57.648144 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:57.625368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.23.130.60:45263)
0808 15:47:57.625511 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.625517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.625533 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:47:57.625582 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.625586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.625665 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.625767 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.628433 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.629906 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.631803 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:57.632853 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:57.633033 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:57.639712 (+ 6679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:47:57.639721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:47:57.639726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:47:57.639765 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:47:57.643974 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.644991 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.644997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.645000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.645244 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:47:57.645730 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:47:57.645735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:47:57.645739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:47:57.646193 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:47:57.646374 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:57.647099 (+ 725us) server_negotiation.cc:299] Negotiation successful
0808 15:47:57.647635 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 15:47:57.648360 16627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:47:57.650103 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.650367 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:57.651033 16744 mvcc.cc:205] Tried to move back new op lower bound from 7316291901910716416 to 7316291901794217984. Current Snapshot: MvccSnapshot[applied={T|T < 7316291901910716416}]
I20260808 15:47:57.652305 16741 catalog_manager.cc:1347] Generated new cluster ID: f510e7384b5442bab77efdd9aeaa7439
I20260808 15:47:57.652442 16741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:47:57.652597 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.652807 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.655925 16619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: Bootstrap starting.
I20260808 15:47:57.658963 16619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: Neither blocks nor log segments found. Creating new log.
I20260808 15:47:57.659948 16619 log.cc:824] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:47:57.665655 16619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: No bootstrap required, opened a new log
I20260808 15:47:57.678654 16619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.679103 16619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c4cb807a824e42a626c4e0752f86d8, State: Initialized, Role: FOLLOWER
I20260808 15:47:57.681519 16619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } }
I20260808 15:47:57.684053 16619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:47:57.684346 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83d1854335074e9694ca52f554f50aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79c4cb807a824e42a626c4e0752f86d8"
I20260808 15:47:57.684832 16593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:47:57.689322 16593 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d1854335074e9694ca52f554f50aa9 in term 1.
I20260808 15:47:57.691083 16591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [term 1 FOLLOWER]: Refusing update from remote peer 83d1854335074e9694ca52f554f50aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:47:57.692406 16707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:47:57.700500 16741 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:47:57.702492 16741 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:47:57.682925 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.704720 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.726240 16741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9: Generated new TSK 0
I20260808 15:47:57.727651 16741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:47:57.731679 16707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.732106 16707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d1854335074e9694ca52f554f50aa9 [sys.catalog]: This master's current role is: LEADER
I20260808 15:47:57.734680 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.734997 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.735275 16755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d1854335074e9694ca52f554f50aa9. Latest consensus state: current_term: 1 leader_uuid: "83d1854335074e9694ca52f554f50aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d1854335074e9694ca52f554f50aa9" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 34205 } } peers { permanent_uuid: "79c4cb807a824e42a626c4e0752f86d8" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 36293 } } peers { permanent_uuid: "62520b8b0ba94fe6bd086a9a4c8a493f" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 45263 } } }
I20260808 15:47:57.735666 16755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:47:57.758457 16769 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260808 15:47:57.821534 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.818106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.23.130.61:36293)
0808 15:47:57.818348 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.818356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.819014 (+ 658us) server_negotiation.cc:407] Connection header received
0808 15:47:57.819344 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.819353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.819475 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.819568 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.821204 (+ 1636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39520: BlockingRecv error: recv got EOF from 127.0.0.1:39520 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 15:47:57.824425 16704 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.817611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.23.130.62:34205)
0808 15:47:57.817858 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.817866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.822493 (+ 4627us) server_negotiation.cc:407] Connection header received
0808 15:47:57.822783 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.822792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.822908 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.822998 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.824166 (+ 1168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58932: BlockingRecv error: recv got EOF from 127.0.0.1:58932 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 15:47:57.826498 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.819322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.23.130.60:45263)
0808 15:47:57.819499 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.819506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.824363 (+ 4857us) server_negotiation.cc:407] Connection header received
0808 15:47:57.824815 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.824821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.824908 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.824984 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.826228 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36916: BlockingRecv error: recv got EOF from 127.0.0.1:36916 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 15:47:57.832445 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.830493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.23.130.61:36293)
0808 15:47:57.830653 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.830657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.831073 (+ 416us) server_negotiation.cc:407] Connection header received
0808 15:47:57.831309 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.831313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.831401 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.831511 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.832169 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39532: BlockingRecv error: recv got EOF from 127.0.0.1:39532 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 15:47:57.838935 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.837701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.23.130.61:36293)
0808 15:47:57.837853 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.837856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.837871 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:47:57.837929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.837932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.837987 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.838063 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.838788 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39540: BlockingRecv error: recv got EOF from 127.0.0.1:39540 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 15:47:57.846086 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.844269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.23.130.61:36293)
0808 15:47:57.844476 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.844482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.844795 (+ 313us) server_negotiation.cc:407] Connection header received
0808 15:47:57.844975 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.844978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.845049 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.845129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.845864 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39554: BlockingRecv error: recv got EOF from 127.0.0.1:39554 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 15:47:57.855355 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.853188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.23.130.61:36293)
0808 15:47:57.853495 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.853501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.853876 (+ 375us) server_negotiation.cc:407] Connection header received
0808 15:47:57.854045 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.854049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.854122 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.854199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.855103 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39560: BlockingRecv error: recv got EOF from 127.0.0.1:39560 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260808 15:47:57.868217 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.866304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.23.130.61:36293)
0808 15:47:57.866555 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.866561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.866886 (+ 325us) server_negotiation.cc:407] Connection header received
0808 15:47:57.867058 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.867062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.867129 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.867215 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.867988 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39574: BlockingRecv error: recv got EOF from 127.0.0.1:39574 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 15:47:57.889707 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.887539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.23.130.61:36293)
0808 15:47:57.887790 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.887799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.888212 (+ 413us) server_negotiation.cc:407] Connection header received
0808 15:47:57.888392 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.888397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.888479 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.888563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.889463 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39582: BlockingRecv error: recv got EOF from 127.0.0.1:39582 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 15:47:57.927640 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.925447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.23.130.61:36293)
0808 15:47:57.925656 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.925663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.926140 (+ 477us) server_negotiation.cc:407] Connection header received
0808 15:47:57.926308 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.926314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.926445 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.926573 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.927373 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39590: BlockingRecv error: recv got EOF from 127.0.0.1:39590 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 15:47:57.997831 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:57.995483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.23.130.61:36293)
0808 15:47:57.995739 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:57.995746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:57.996080 (+ 334us) server_negotiation.cc:407] Connection header received
0808 15:47:57.996350 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:57.996355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:57.996436 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:57.996533 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:57.997560 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39604: BlockingRecv error: recv got EOF from 127.0.0.1:39604 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260808 15:47:58.131822 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:58.129703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.23.130.61:36293)
0808 15:47:58.129979 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.129987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:58.130328 (+ 341us) server_negotiation.cc:407] Connection header received
0808 15:47:58.130603 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:58.130608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:58.130680 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:58.130793 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:58.131621 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39608: BlockingRecv error: recv got EOF from 127.0.0.1:39608 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260808 15:47:58.393994 16624 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:47:58.392190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.23.130.61:36293)
0808 15:47:58.392436 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.392443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:58.392662 (+ 219us) server_negotiation.cc:407] Connection header received
0808 15:47:58.392826 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:58.392830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:58.392902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:58.393043 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:47:58.393827 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39618: BlockingRecv error: recv got EOF from 127.0.0.1:39618 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:47:58.626972 16746 catalog_manager.cc:1259] Loaded cluster ID: f510e7384b5442bab77efdd9aeaa7439
I20260808 15:47:58.627128 16746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: loading cluster ID for follower catalog manager: success
I20260808 15:47:58.629745 16746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: acquiring CA information for follower catalog manager: success
I20260808 15:47:58.631681 16746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62520b8b0ba94fe6bd086a9a4c8a493f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:47:58.763523 16769 catalog_manager.cc:1259] Loaded cluster ID: f510e7384b5442bab77efdd9aeaa7439
I20260808 15:47:58.763680 16769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: loading cluster ID for follower catalog manager: success
I20260808 15:47:58.767622 16769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: acquiring CA information for follower catalog manager: success
I20260808 15:47:58.770177 16769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79c4cb807a824e42a626c4e0752f86d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:47:58.946069 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:58.924394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.23.130.61:36293)
0808 15:47:58.937058 (+ 12664us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.937066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:58.937759 (+ 693us) server_negotiation.cc:407] Connection header received
0808 15:47:58.937992 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:58.937998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:58.938248 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:58.938462 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:58.939039 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.940550 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.942576 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.943388 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.943551 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:58.944246 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:58.945222 (+ 976us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:58.945417 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:58.945491 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 15:47:58.945621 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12507,"thread_start_us":12252,"threads_started":1}
I20260808 15:47:58.976387 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:58.955049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.23.130.60:45263)
0808 15:47:58.967451 (+ 12402us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.967459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:58.968218 (+ 759us) server_negotiation.cc:407] Connection header received
0808 15:47:58.968439 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:58.968446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:58.968707 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:58.968960 (+ 253us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:58.969545 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.971134 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.973237 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.973979 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.974114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:58.974784 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:58.975579 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:58.975736 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:58.975864 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:47:58.975962 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12234,"thread_start_us":12416,"threads_started":1}
I20260808 15:47:59.001152 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:58.985246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.23.130.61:36293)
0808 15:47:58.985433 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.985440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:58.986954 (+ 1514us) server_negotiation.cc:407] Connection header received
0808 15:47:58.987254 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:58.987266 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:58.987379 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:58.987457 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:58.988331 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.989576 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.991573 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:58.992235 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:58.992331 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:58.999861 (+ 7530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:59.000083 (+ 222us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:59.000170 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:59.000629 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 15:47:59.000740 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:47:59.011224 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:58.987456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.23.130.60:45263)
0808 15:47:58.987717 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:58.987722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:59.002146 (+ 14424us) server_negotiation.cc:407] Connection header received
0808 15:47:59.002462 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:59.002469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:59.002597 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:59.002684 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:59.003870 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:59.005314 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:59.008620 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:59.009401 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:59.009509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:59.010193 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:59.010451 (+ 258us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:59.010524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:59.010767 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 15:47:59.010874 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260808 15:47:59.015725 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:47:58.992531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.23.130.62:34205)
0808 15:47:59.007540 (+ 15009us) server_negotiation.cc:206] Beginning negotiation
0808 15:47:59.007546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:47:59.007569 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:47:59.007690 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:47:59.007696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:47:59.007965 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:47:59.009095 (+ 1130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:47:59.009320 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:59.010582 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:59.012657 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:47:59.013311 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:47:59.014006 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:47:59.014205 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:47:59.014701 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:47:59.015274 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0808 15:47:59.015327 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 15:47:59.015414 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14856,"thread_start_us":15935,"threads_started":1}
I20260808 15:47:59.022764 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16466
I20260808 15:47:59.024078 16535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:59.146571 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16466
W20260808 15:47:59.174942 16647 connection.cc:535] server connection from 127.0.0.1:36874 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:47:59.175457 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16549
I20260808 15:47:59.176662 16613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:59.284413 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16549
I20260808 15:47:59.307600 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16627
I20260808 15:47:59.308787 16693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:47:59.416112 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16627
2026-08-08T15:47:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5068 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:47:59 dist-test-slave-gp6n krb5kdc[16870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:47:59 dist-test-slave-gp6n krb5kdc[16870](info): set up 2 sockets
Aug 08 15:47:59 dist-test-slave-gp6n krb5kdc[16870](info): commencing operation
krb5kdc: starting...
W20260808 15:48:00.620819 24072 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/test-user.keytab.
Aug 08 15:48:00 dist-test-slave-gp6n krb5kdc[16870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
Entry for principal kudu/127.23.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab.
I20260808 15:48:00.953462 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37105
--webserver_interface=127.23.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:01.300573 16885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:48:01.300900 16885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:01.300992 16885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:01.301074 16885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:01.308669 16885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:01.308774 16885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:01.308835 16885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:01.308883 16885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:01.308948 16885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:01.318879 16885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.62.keytab
--principal=kudu/127.23.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.62:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:01.320143 16885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:01.321509 16885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:01.334311 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:01.334277 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:01.334919 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:01.335355 16885 server_base.cc:1034] running on GCE node
I20260808 15:48:01.336251 16885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:01.337706 16885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:01.338951 16885 hybrid_clock.cc:648] HybridClock initialized: now 1786204081338875 us; error 51 us; skew 500 ppm
I20260808 15:48:01.339288 16885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:01 dist-test-slave-gp6n krb5kdc[16870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:01.347563 16885 init.cc:377] Logged in from keytab as kudu/127.23.130.62@KRBTEST.COM (short username kudu)
I20260808 15:48:01.349185 16885 webserver.cc:460] Webserver started at http://127.23.130.62:33273/ using document root <none> and password file <none>
I20260808 15:48:01.349918 16885 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:01.350064 16885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:01.350464 16885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:01.353492 16885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data/instance:
uuid: "94bcfabed08a49019d80bd8e11ea4a07"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "09c5e51521f48aa457e12561650f291b"
server_key_iv: "85f064eaae12a0d81c717a7508dce15e"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.354310 16885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance:
uuid: "94bcfabed08a49019d80bd8e11ea4a07"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "09c5e51521f48aa457e12561650f291b"
server_key_iv: "85f064eaae12a0d81c717a7508dce15e"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.359668 16885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:48:01.363386 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:01.364766 16885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:48:01.364944 16885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal
uuid: "94bcfabed08a49019d80bd8e11ea4a07"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "09c5e51521f48aa457e12561650f291b"
server_key_iv: "85f064eaae12a0d81c717a7508dce15e"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.365172 16885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:01.393592 16885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:01.397833 16885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:01.398124 16885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:01.417948 16885 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.62:37105
I20260808 15:48:01.417999 16957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.62:37105 every 8 connection(s)
I20260808 15:48:01.419373 16885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:01.423483 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:01.426072 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16885
I20260808 15:48:01.426277 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-0/wal/instance
I20260808 15:48:01.426757 24072 external_mini_cluster.cc:1428] Setting key 23efcf3f0bdea08e7dcb0f4b4f250331
I20260808 15:48:01.430943 16958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:48:01 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:48:01.455825 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.434473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37105 (local address 127.0.0.1:49770)
0808 15:48:01.435301 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.435353 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:01.437676 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:01.438648 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:01.438655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:01.439425 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:01.439784 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.439803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.441504 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.441511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.442763 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.442775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.443797 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.443805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.444031 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:01.444763 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:01.444786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:01.447374 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:01.451074 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:01.451086 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:01.451099 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:01.451534 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:01.451980 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:01.451987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:01.451991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:01.452163 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:01.452988 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:01.453003 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:01.453246 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 15:48:01.453528 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 15:48:01.454206 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":166,"thread_start_us":349,"threads_started":1}
I20260808 15:48:01.455842 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.435779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.23.130.62:37105)
0808 15:48:01.436587 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:01.436596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:01.436660 (+ 64us) server_negotiation.cc:407] Connection header received
0808 15:48:01.438075 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:01.438081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:01.438344 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:01.438621 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:01.440026 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.441336 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.442932 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.443532 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.443745 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:01.447648 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:01.447657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:01.447668 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:01.447698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:01.450919 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:01.451690 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:01.451697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:01.451700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:01.451859 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:01.452291 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:01.452302 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:01.452305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:01.452840 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:01.453078 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:01.453664 (+ 586us) server_negotiation.cc:299] Negotiation successful
0808 15:48:01.454069 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":584,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for kudu/127.23.130.61@KRBTEST.COM; defaulting to no policy
I20260808 15:48:01.459596 16958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:01.461205 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.460525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34056)
0808 15:48:01.460731 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.461032 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.61@KRBTEST.COM" created.
W20260808 15:48:01.461652 16905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
I20260808 15:48:01.463718 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.463265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34066)
0808 15:48:01.463450 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.463544 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 15:48:01.464103 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39109: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
Entry for principal kudu/127.23.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab.
I20260808 15:48:01.502470 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:39109
--webserver_interface=127.23.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:48:01.513558 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } attempt: 1
I20260808 15:48:01.514932 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.514368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34076)
0808 15:48:01.514648 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.514783 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260808 15:48:01.516335 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.515842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34084)
0808 15:48:01.516025 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.516138 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:48:01.516602 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39109: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
I20260808 15:48:01.584889 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } attempt: 2
I20260808 15:48:01.586391 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.585804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34088)
0808 15:48:01.586085 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.586180 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 15:48:01.587524 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.587145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34096)
0808 15:48:01.587321 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.587391 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:48:01.587716 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39109: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
I20260808 15:48:01.678992 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } attempt: 3
I20260808 15:48:01.680225 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.679755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34098)
0808 15:48:01.679982 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.680090 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 15:48:01.681233 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.680888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34112)
0808 15:48:01.681034 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.681105 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:48:01.681402 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39109: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
I20260808 15:48:01.824640 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } attempt: 4
I20260808 15:48:01.826012 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.825331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34126)
0808 15:48:01.825706 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.825829 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260808 15:48:01.827224 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.826777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34134)
0808 15:48:01.826953 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.827062 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 15:48:01.827481 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.61:39109: Network error: Client connection negotiation failed: client connection to 127.23.130.61:39109: connect: Connection refused (error 111)
W20260808 15:48:01.853194 16965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:48:01.853533 16965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:01.853600 16965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:01.853684 16965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:01.860988 16965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:01.861089 16965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:01.861147 16965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:01.861196 16965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:01.861259 16965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:01.871451 16965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.61.keytab
--principal=kudu/127.23.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.61:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:01.872700 16965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:01.873976 16965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:01.885789 16970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:01.885748 16971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:01.885859 16973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:01.886884 16965 server_base.cc:1034] running on GCE node
I20260808 15:48:01.887594 16965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:01.888893 16965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:01.890144 16965 hybrid_clock.cc:648] HybridClock initialized: now 1786204081890064 us; error 52 us; skew 500 ppm
I20260808 15:48:01.890575 16965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:01 dist-test-slave-gp6n krb5kdc[16870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:01.893962 16965 init.cc:377] Logged in from keytab as kudu/127.23.130.61@KRBTEST.COM (short username kudu)
I20260808 15:48:01.895448 16965 webserver.cc:460] Webserver started at http://127.23.130.61:40107/ using document root <none> and password file <none>
I20260808 15:48:01.896130 16965 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:01.896265 16965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:01.896598 16965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:01.899606 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data/instance:
uuid: "e75f5b79ed5c4ebdaff477c878dedd68"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "deb6a8923726b9579cbc1f5224d1c438"
server_key_iv: "848638fd86eeee6a040ecdfa985f6d66"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.900439 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal/instance:
uuid: "e75f5b79ed5c4ebdaff477c878dedd68"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "deb6a8923726b9579cbc1f5224d1c438"
server_key_iv: "848638fd86eeee6a040ecdfa985f6d66"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.905426 16965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 15:48:01.908779 16981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:01.910156 16965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:48:01.910390 16965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal
uuid: "e75f5b79ed5c4ebdaff477c878dedd68"
format_stamp: "Formatted at 2026-08-08 15:48:01 on dist-test-slave-gp6n"
server_key: "deb6a8923726b9579cbc1f5224d1c438"
server_key_iv: "848638fd86eeee6a040ecdfa985f6d66"
server_key_version: "encryptionkey@0"
I20260808 15:48:01.910671 16965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:01.931072 16965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:01.935745 16965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:01.936030 16965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:01.955273 16965 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.61:39109
I20260808 15:48:01.955292 17036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.61:39109 every 8 connection(s)
I20260808 15:48:01.956799 16965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/data/info.pb
I20260808 15:48:01.960469 17038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:01.967147 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 16965
I20260808 15:48:01.967340 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-1/wal/instance
I20260808 15:48:01.967903 24072 external_mini_cluster.cc:1428] Setting key f49c82b81d0c937db69635780efbee12
I20260808 15:48:01.967105 17038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:48:01 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:48:01.989786 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.970335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.23.130.62:37105)
0808 15:48:01.970895 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:01.970903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:01.971940 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 15:48:01.973863 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:01.973879 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:01.973974 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:01.974109 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:01.975753 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.976886 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.978756 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.979473 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.979614 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:01.983161 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:01.983171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:01.983177 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:01.983239 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:01.986180 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:01.987187 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:01.987192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:01.987194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:01.987336 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:01.987860 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:01.987863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:01.987865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:01.988151 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:01.988289 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:01.989263 (+ 974us) server_negotiation.cc:299] Negotiation successful
0808 15:48:01.989496 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260808 15:48:01.991508 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.970956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37105 (local address 127.0.0.1:49786)
0808 15:48:01.971748 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.971795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:01.973588 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:01.974306 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:01.974314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:01.974972 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:01.975527 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.975545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.977119 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.977126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.978614 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.978628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.979699 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.979704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.979915 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:01.980623 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:01.980647 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:01.982900 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:01.986322 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:01.986333 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:01.986351 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:01.987042 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:01.987463 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:01.987471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:01.987477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:01.987742 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:01.988287 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:01.988297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:01.988781 (+ 484us) client_negotiation.cc:769] Sending connection context
0808 15:48:01.989174 (+ 393us) client_negotiation.cc:240] Negotiation successful
0808 15:48:01.989778 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":349,"threads_started":1}
I20260808 15:48:01.993340 17038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:48:01 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.130.60@KRBTEST.COM" created.
I20260808 15:48:02.008349 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.994215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34148)
0808 15:48:01.994382 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:01.994438 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:01.994536 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:01.995525 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:01.995529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:01.995971 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:01.996095 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.996103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.997492 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.997498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.998361 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:01.998371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.999243 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.999249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:01.999423 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.000094 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.000110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.002377 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.005738 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.005745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.005749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.006135 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.006561 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.006565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.006569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.006711 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.007322 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.007327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.007473 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.007626 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.007915 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:48:02.008410 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:01.994452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.23.130.61:39109)
0808 15:48:01.995112 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:01.995118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:01.995141 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:48:01.995181 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:01.995187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:01.995352 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:01.995515 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:01.996261 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.997352 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.998540 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:01.999081 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:01.999226 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.002634 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.002645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.002658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.002686 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.005618 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.006275 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.006283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.006285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.006391 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.006829 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.006834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.006837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.007202 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.007447 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.007779 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.008033 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":518,"thread_start_us":372,"threads_started":1}
I20260808 15:48:02.011068 17038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:02.012177 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.011731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49506)
0808 15:48:02.011896 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.012025 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:48:02.012408 16986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.014959 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.014628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49516)
0808 15:48:02.014776 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.014852 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 15:48:02.015226 17038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
Entry for principal kudu/127.23.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab.
I20260808 15:48:02.046986 24072 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2oCF2B/build/asan/bin/kudu
/tmp/dist-test-task2oCF2B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:36019
--webserver_interface=127.23.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.130.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:48:02.064581 17038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 1
I20260808 15:48:02.065943 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.065311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49532)
0808 15:48:02.065566 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.065707 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 15:48:02.067193 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.066743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49540)
0808 15:48:02.066936 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.067041 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:48:02.067449 17038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.126777 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } attempt: 5
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:48:02.135706 17038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 2
I20260808 15:48:02.136968 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.136318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49550)
0808 15:48:02.136603 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.136724 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 15:48:02.138469 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.137943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49558)
0808 15:48:02.138131 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.138231 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 15:48:02.138799 17038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.142930 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.127743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.23.130.61:39109)
0808 15:48:02.127937 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.127942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.128007 (+ 65us) server_negotiation.cc:407] Connection header received
0808 15:48:02.128218 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.128223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.128312 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.128414 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.129500 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.130213 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.131987 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.132469 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.132575 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.136501 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.136509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.136514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.136554 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.140095 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.140837 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.140843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.140849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.140986 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.141395 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.141402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.141405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.141703 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.141875 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.142327 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.142589 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:48:02.142973 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.127627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34158)
0808 15:48:02.127883 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.127911 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.128070 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.128488 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.128493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.129034 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.129320 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.129334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.130353 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.130367 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.131821 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.131839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.132638 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.132646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.132854 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.133547 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.133571 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.136278 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.140246 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.140253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.140257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.140691 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.141111 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.141117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.141121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.141273 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.141845 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.141850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.142005 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.142202 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.142560 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 15:48:02.144025 16958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:02.145037 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.144648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49560)
0808 15:48:02.144829 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.144903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:48:02.146111 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.145734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49576)
0808 15:48:02.145886 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.145988 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:48:02.146341 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.197645 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 1
I20260808 15:48:02.199074 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.198531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49588)
0808 15:48:02.198754 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.198868 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:48:02.200209 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.199789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49596)
0808 15:48:02.199975 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.200075 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 15:48:02.200525 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.230044 17038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 3
I20260808 15:48:02.231308 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.230754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49606)
0808 15:48:02.230986 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.231108 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:48:02.232435 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.231957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49622)
0808 15:48:02.232196 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.232285 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260808 15:48:02.232677 17038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.268779 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 2
I20260808 15:48:02.270274 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.269560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49634)
0808 15:48:02.269870 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.270016 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 15:48:02.271427 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.271035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49638)
0808 15:48:02.271195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.271287 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 15:48:02.271672 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.375931 17038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 4
I20260808 15:48:02.377224 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.376645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49654)
0808 15:48:02.376917 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.377031 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 15:48:02.377900 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 3
I20260808 15:48:02.378504 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.378051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49656)
0808 15:48:02.378259 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.378323 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260808 15:48:02.378762 17038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
I20260808 15:48:02.379153 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.378513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49666)
0808 15:48:02.378897 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.378992 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260808 15:48:02.380142 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.379824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49676)
0808 15:48:02.379990 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.380056 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 15:48:02.380331 16958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.130.60:36019: Network error: Client connection negotiation failed: client connection to 127.23.130.60:36019: connect: Connection refused (error 111)
W20260808 15:48:02.400074 17046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:48:02.400444 17046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:02.400574 17046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:02.400717 17046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:02.407768 17046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:02.407886 17046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:02.408010 17046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:02.408107 17046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:02.408203 17046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:02.418367 17046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.130.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.130.62:37105,127.23.130.61:39109,127.23.130.60:36019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/krb5kdc/kudu_127.23.130.60.keytab
--principal=kudu/127.23.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.130.60:36019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:02.419723 17046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:02.421034 17046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:02.432857 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:02.433111 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:02.433229 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:02.433926 17046 server_base.cc:1034] running on GCE node
I20260808 15:48:02.434751 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:02.436075 17046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:02.437342 17046 hybrid_clock.cc:648] HybridClock initialized: now 1786204082437283 us; error 46 us; skew 500 ppm
I20260808 15:48:02.437731 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:02.441294 17046 init.cc:377] Logged in from keytab as kudu/127.23.130.60@KRBTEST.COM (short username kudu)
I20260808 15:48:02.442764 17046 webserver.cc:460] Webserver started at http://127.23.130.60:45639/ using document root <none> and password file <none>
I20260808 15:48:02.443413 17046 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:02.443502 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:02.443779 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:02.446879 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data/instance:
uuid: "ccd0a11e5a564253903b5ea3688c5c98"
format_stamp: "Formatted at 2026-08-08 15:48:02 on dist-test-slave-gp6n"
server_key: "aba12c75e71dcfef22e8d3dbed5daad7"
server_key_iv: "8f021b03c43e4ca3d8db1435f2698bd7"
server_key_version: "encryptionkey@0"
I20260808 15:48:02.447664 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal/instance:
uuid: "ccd0a11e5a564253903b5ea3688c5c98"
format_stamp: "Formatted at 2026-08-08 15:48:02 on dist-test-slave-gp6n"
server_key: "aba12c75e71dcfef22e8d3dbed5daad7"
server_key_iv: "8f021b03c43e4ca3d8db1435f2698bd7"
server_key_version: "encryptionkey@0"
I20260808 15:48:02.452893 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:48:02.456418 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:02.457763 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:02.457906 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data,/tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal
uuid: "ccd0a11e5a564253903b5ea3688c5c98"
format_stamp: "Formatted at 2026-08-08 15:48:02 on dist-test-slave-gp6n"
server_key: "aba12c75e71dcfef22e8d3dbed5daad7"
server_key_iv: "8f021b03c43e4ca3d8db1435f2698bd7"
server_key_version: "encryptionkey@0"
I20260808 15:48:02.458091 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:02.509881 17046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:02.515195 17046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:02.515470 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:02.536141 17046 rpc_server.cc:307] RPC server started. Bound to: 127.23.130.60:36019
I20260808 15:48:02.536162 17118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.130.60:36019 every 8 connection(s)
I20260808 15:48:02.537565 17046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/data/info.pb
I20260808 15:48:02.541229 17119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:02.542395 24072 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2oCF2B/build/asan/bin/kudu as pid 17046
I20260808 15:48:02.542613 24072 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2oCF2B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786203706771851-24072-0/minicluster-data/master-2/wal/instance
I20260808 15:48:02.543179 24072 external_mini_cluster.cc:1428] Setting key 818b065fcd37e5c508c2f9f1c77780fd
I20260808 15:48:02.550848 17119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:02.557547 16958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 4
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.62@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
I20260808 15:48:02.568812 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.545140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.23.130.60:36019)
0808 15:48:02.546017 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.546032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.546071 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:48:02.547732 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.547754 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.548170 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.548498 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.550237 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.551689 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.553353 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.554259 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.554480 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.559534 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.559544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.559561 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.559631 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.564169 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.564947 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.564957 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.564966 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.565169 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.565597 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.565605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.565609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.566302 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.566648 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.567099 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.567528 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":411,"threads_started":1}
I20260808 15:48:02.571641 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.553468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.23.130.62:37105)
0808 15:48:02.553973 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.553979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.554691 (+ 712us) server_negotiation.cc:407] Connection header received
0808 15:48:02.554769 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.554776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.554865 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.554996 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.556199 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.557359 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.559436 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.559978 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.560125 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.564149 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.564156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.564162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.564210 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.567963 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.568838 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.568845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.568848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.569000 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.569505 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.569512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.569518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.569851 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.570004 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.570998 (+ 994us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.571227 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260808 15:48:02.572623 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.553582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.62:37105 (local address 127.0.0.1:49796)
0808 15:48:02.554439 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.554487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.554630 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.555097 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.555104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.555784 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.556023 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.556031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.557561 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.557568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.559249 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.559261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.560169 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.560177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.560390 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.561000 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.561021 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.563939 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.568102 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.568115 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.568124 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.568689 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.569152 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.569159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.569163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.569355 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.570003 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.570011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.570276 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.571836 (+ 1560us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.572174 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":454,"threads_started":1}
I20260808 15:48:02.574090 17119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:02.575404 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.558195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49702)
0808 15:48:02.558550 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.558570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.558678 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.559550 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.559555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.560137 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.560413 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.560429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.562368 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.562375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.563657 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.563671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.564666 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.564673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.564911 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.565678 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.565694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.568440 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.572547 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.572555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.572561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.573044 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.573522 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.573529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.573533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.573688 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.574361 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.574366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.574589 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.574744 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.575003 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260808 15:48:02.576333 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.558391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.23.130.60:36019)
0808 15:48:02.559223 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.559228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.559251 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:48:02.559298 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.559306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.559390 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.559497 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.560594 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.562219 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.563805 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.564498 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.564625 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.568705 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.568714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.568726 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.568763 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.572395 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.573205 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.573212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.573217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.573377 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.573848 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.573854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.573860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.574231 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.574390 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.575905 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.576092 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":491,"threads_started":1}
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.62@KRBTEST.COM
I20260808 15:48:02.583547 16958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: Bootstrap starting.
I20260808 15:48:02.587957 16958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:02.589464 16958 log.cc:824] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:02.589774 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.572323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.23.130.62:37105)
0808 15:48:02.572508 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.572516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.572534 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:02.572838 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.572845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.572923 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.573031 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.574599 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.575728 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.577565 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.577986 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.578139 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.583747 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.583752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.583755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.583784 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.586836 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.587486 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.587498 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.587502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.587621 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.588081 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.588088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.588092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.588539 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.588853 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.589147 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.589410 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:48:02.589927 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.574828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.61:39109 (local address 127.0.0.1:34172)
0808 15:48:02.575054 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.575073 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.575161 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.575470 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.575475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.575936 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.576133 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.576144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.577907 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.577912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.579243 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.579254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.579977 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.579983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.580171 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.580709 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.580725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.583097 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.587064 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.587069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.587073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.587594 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.588109 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.588114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.588117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.588296 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.588989 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.589013 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.589168 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.589326 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.589581 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260808 15:48:02.590866 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.575009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.23.130.61:39109)
0808 15:48:02.575183 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.575191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.575210 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:48:02.575269 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.575274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.575347 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.575455 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.576324 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.577375 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.579398 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.579820 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.579952 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.583312 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.583318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.583324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.583366 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.586890 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.587736 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.587741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.587746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.587930 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.588422 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.588427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.588431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.588778 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.588927 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.590200 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.590461 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:48:02.591837 17119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:48:02.594568 16958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: No bootstrap required, opened a new log
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.60@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.130.61@KRBTEST.COM
I20260808 15:48:02.604782 16958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.605175 16958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94bcfabed08a49019d80bd8e11ea4a07, State: Initialized, Role: FOLLOWER
I20260808 15:48:02.605741 16958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.606791 17128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.607199 17128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.607963 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.592574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.23.130.60:36019)
0808 15:48:02.592715 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.592719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.592735 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:48:02.592847 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.592852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.592927 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.593023 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.593659 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.594789 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.596459 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.597029 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.597169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.600151 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.600157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.600162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.600197 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.603991 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.604814 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.604821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.604826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.604978 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.605394 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.605401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.605404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.605830 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.605978 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.607408 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.607619 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:48:02.608366 16958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:02.608408 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.592504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49712)
0808 15:48:02.592637 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.592654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.592741 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.593037 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.593041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.593403 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.593533 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.593540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.594930 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.594939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.596301 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.596316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.597183 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.597189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.597450 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.598016 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.598032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.599978 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.604130 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.604134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.604137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.604670 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.605109 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.605113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.605116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.605288 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.606144 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.606149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.606317 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.606525 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.606877 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 15:48:02.608464 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.23.130.61:39109)
0808 15:48:02.591977 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.591981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.591994 (+ 13us) server_negotiation.cc:407] Connection header received
0808 15:48:02.592052 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.592056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.592119 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.592201 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.593809 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.594869 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.596073 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.596591 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.596708 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.602130 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.602135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.602138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.602168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.605317 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.606166 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.606173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.606179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.606389 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.606910 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.606914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.606917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.607260 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.607577 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.607876 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.608102 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:48:02.614114 24072 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:02.614142 17119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: Bootstrap starting.
I20260808 15:48:02.617442 17119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:02.618336 16933 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "94bcfabed08a49019d80bd8e11ea4a07"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49812
I20260808 15:48:02.618744 17119 log.cc:824] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:02.618858 16933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 15:48:02.619107 16933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:02.623729 16933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.625586 17119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: No bootstrap required, opened a new log
I20260808 15:48:02.626520 16933 leader_election.cc:288] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [CANDIDATE]: Term 1 election: Requested vote from peers e75f5b79ed5c4ebdaff477c878dedd68 (127.23.130.61:39109), ccd0a11e5a564253903b5ea3688c5c98 (127.23.130.60:36019)
I20260808 15:48:02.637877 17119 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.638475 17119 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd0a11e5a564253903b5ea3688c5c98, State: Initialized, Role: FOLLOWER
I20260808 15:48:02.639094 17119 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.645721 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.646251 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.647035 17119 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:02.647115 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94bcfabed08a49019d80bd8e11ea4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccd0a11e5a564253903b5ea3688c5c98"
I20260808 15:48:02.647588 17093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [term 0 FOLLOWER]: Advancing to term 1
W20260808 15:48:02.648444 17141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:48:02.648576 17141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 15:48:02.651085 16908 tablet.cc:2369] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:02.655076 17093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bcfabed08a49019d80bd8e11ea4a07 in term 1.
I20260808 15:48:02.656270 16905 leader_election.cc:302] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94bcfabed08a49019d80bd8e11ea4a07, ccd0a11e5a564253903b5ea3688c5c98; no voters:
I20260808 15:48:02.656875 17128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:02.673306 17128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [term 1 LEADER]: Becoming Leader. State: Replica: 94bcfabed08a49019d80bd8e11ea4a07, State: Running, Role: LEADER
I20260808 15:48:02.674819 17128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.631027 16885 master_runner.cc:186] Error getting master registration for 127.23.130.61:39109: OK, instance_id { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" instance_seqno: 1786204081949848 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:48:02.677979 17038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } attempt: 5
I20260808 15:48:02.678910 16885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 08 15:48:02 dist-test-slave-gp6n krb5kdc[16870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.130.61@KRBTEST.COM for kudu/127.23.130.60@KRBTEST.COM
I20260808 15:48:02.691759 17157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94bcfabed08a49019d80bd8e11ea4a07. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.692325 17157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:02.694497 17163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:02.697513 17163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:02.703781 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.678877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.23.130.60:36019)
0808 15:48:02.679101 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.679109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.679956 (+ 847us) server_negotiation.cc:407] Connection header received
0808 15:48:02.680040 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.680048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.680147 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.680253 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.681750 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.683291 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.685445 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.686217 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.686349 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.692148 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.692154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.692159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.692195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.697183 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.698559 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.698565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.698595 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.698889 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.699771 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.699777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.699809 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.700586 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.700857 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.701769 (+ 912us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.702215 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:48:02.704668 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.679310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.130.60:36019 (local address 127.0.0.1:49732)
0808 15:48:02.679667 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:02.679690 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:02.679829 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:02.680508 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:02.680515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:02.681220 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:02.681512 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.681528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.683513 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.683523 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.685268 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:02.685286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.686437 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.686444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:02.686867 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.687725 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:02.687772 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:02.691888 (+ 4116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:02.697453 (+ 5565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.697463 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.697467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.698346 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.699165 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:02.699199 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:02.699208 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:02.699492 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:02.700993 (+ 1501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:02.701000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:02.701276 (+ 276us) client_negotiation.cc:769] Sending connection context
0808 15:48:02.702262 (+ 986us) client_negotiation.cc:240] Negotiation successful
0808 15:48:02.703450 (+ 1188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260808 15:48:02.712369 17038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: Bootstrap starting.
I20260808 15:48:02.716658 17038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:02.718017 17038 log.cc:824] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:02.718743 17093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [term 1 FOLLOWER]: Refusing update from remote peer 94bcfabed08a49019d80bd8e11ea4a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:02.720065 17157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:02.728206 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.652562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.23.130.61:39109)
0808 15:48:02.652780 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.652788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.664119 (+ 11331us) server_negotiation.cc:407] Connection header received
0808 15:48:02.667411 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.667422 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.667543 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.667638 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.686588 (+ 18950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.688063 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.693671 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.694661 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.694775 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.709699 (+ 14924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.709707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.709711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.709753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.713495 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.721859 (+ 8364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.721865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.721870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.722074 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.724648 (+ 2574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.724654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.724658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.725123 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.725435 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.727222 (+ 1787us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.727650 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:48:02.739589 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.652138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.23.130.62:37105)
0808 15:48:02.652361 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.652371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.670113 (+ 17742us) server_negotiation.cc:407] Connection header received
0808 15:48:02.670783 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.670793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.670899 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.670988 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.690175 (+ 19187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.691438 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.697343 (+ 5905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.698104 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.698221 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.704621 (+ 6400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.704630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.704637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.704683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.708773 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.722279 (+ 13506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.722288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.722294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.722499 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.723391 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.723397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.723403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.723882 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.724226 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.738711 (+ 14485us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.739099 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:48:02.745697 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:02.664317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.23.130.60:36019)
0808 15:48:02.664537 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.664546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:02.685179 (+ 20633us) server_negotiation.cc:407] Connection header received
0808 15:48:02.685366 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:02.685372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:02.685461 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:02.685556 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:02.699477 (+ 13921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.700978 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.709677 (+ 8699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:02.710650 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:02.710775 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:02.716568 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:02.716576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:02.716580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:02.716623 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:02.720799 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.725993 (+ 5194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.726002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.726008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.726210 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:02.741059 (+ 14849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:02.741070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:02.741076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:02.741562 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:02.741954 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:02.744859 (+ 2905us) server_negotiation.cc:299] Negotiation successful
0808 15:48:02.745181 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:48:02.747413 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94bcfabed08a49019d80bd8e11ea4a07. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.747843 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.756583 17164 mvcc.cc:205] Tried to move back new op lower bound from 7316291922800381952 to 7316291922656595968. Current Snapshot: MvccSnapshot[applied={T|T < 7316291922800381952}]
I20260808 15:48:02.757221 17038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: No bootstrap required, opened a new log
I20260808 15:48:02.757730 17157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.758014 17157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:02.759903 17163 catalog_manager.cc:1347] Generated new cluster ID: ac466c08448c4f27b818f36548c92050
I20260808 15:48:02.759999 17163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:02.760509 17168 mvcc.cc:205] Tried to move back new op lower bound from 7316291922800381952 to 7316291922656595968. Current Snapshot: MvccSnapshot[applied={T|T < 7316291922800381952}]
I20260808 15:48:02.767704 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.767985 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.773996 17038 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.774448 17038 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75f5b79ed5c4ebdaff477c878dedd68, State: Initialized, Role: FOLLOWER
I20260808 15:48:02.774979 17038 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } }
I20260808 15:48:02.790478 17175 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.790895 17175 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.791867 17038 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:02.793049 17011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94bcfabed08a49019d80bd8e11ea4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e75f5b79ed5c4ebdaff477c878dedd68"
I20260808 15:48:02.793535 17011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:02.797060 17011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bcfabed08a49019d80bd8e11ea4a07 in term 1.
I20260808 15:48:02.797781 17010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [term 1 FOLLOWER]: Refusing update from remote peer 94bcfabed08a49019d80bd8e11ea4a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:02.799513 17128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:02.789986 17046 master_runner.cc:186] Error getting master registration for 127.23.130.61:39109: OK, instance_id { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" instance_seqno: 1786204081949848 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:48:02.804912 17046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:48:02.808441 17174 catalog_manager.cc:1259] Loaded cluster ID: ac466c08448c4f27b818f36548c92050
I20260808 15:48:02.808542 17174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: loading cluster ID for follower catalog manager: success
W20260808 15:48:02.810102 17174 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 15:48:02.810263 17174 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:48:02.820638 17163 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:02.822732 17163 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:02.834059 17175 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94bcfabed08a49019d80bd8e11ea4a07. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.834434 17175 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.834862 17181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.835220 17181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:48:02.836549 17157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94bcfabed08a49019d80bd8e11ea4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bcfabed08a49019d80bd8e11ea4a07" member_type: VOTER last_known_addr { host: "127.23.130.62" port: 37105 } } peers { permanent_uuid: "e75f5b79ed5c4ebdaff477c878dedd68" member_type: VOTER last_known_addr { host: "127.23.130.61" port: 39109 } } peers { permanent_uuid: "ccd0a11e5a564253903b5ea3688c5c98" member_type: VOTER last_known_addr { host: "127.23.130.60" port: 36019 } } }
I20260808 15:48:02.836841 17157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:02.855201 17163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94bcfabed08a49019d80bd8e11ea4a07: Generated new TSK 0
I20260808 15:48:02.856384 17163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:48:02.897375 17194 catalog_manager.cc:1259] Loaded cluster ID: ac466c08448c4f27b818f36548c92050
I20260808 15:48:02.897491 17194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: loading cluster ID for follower catalog manager: success
I20260808 15:48:02.902026 17194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: acquiring CA information for follower catalog manager: success
I20260808 15:48:02.904385 17194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e75f5b79ed5c4ebdaff477c878dedd68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 15:48:03.009805 17043 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.000455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.23.130.61:39109)
0808 15:48:03.000664 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.000670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.006118 (+ 5448us) server_negotiation.cc:407] Connection header received
0808 15:48:03.007781 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.007788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.007898 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.007975 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.009555 (+ 1580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34200: BlockingRecv error: recv got EOF from 127.0.0.1:34200 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:48:03.011372 17123 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:02.991831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.23.130.62:37105)
0808 15:48:02.992064 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:02.992071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.005606 (+ 13535us) server_negotiation.cc:407] Connection header received
0808 15:48:03.008191 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.008197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.008278 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.008368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.011111 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49842: BlockingRecv error: recv got EOF from 127.0.0.1:49842 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:48:03.013978 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.006706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.23.130.60:36019)
0808 15:48:03.006890 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.006896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.012043 (+ 5147us) server_negotiation.cc:407] Connection header received
0808 15:48:03.012635 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.012641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.012726 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.012800 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.013714 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49742: BlockingRecv error: recv got EOF from 127.0.0.1:49742 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:48:03.063453 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.047462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.23.130.61:39109)
0808 15:48:03.047652 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.047659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.054850 (+ 7191us) server_negotiation.cc:407] Connection header received
0808 15:48:03.055440 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.055447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.055689 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.055841 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.056682 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.057878 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.060035 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.060745 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.060837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.061766 (+ 929us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.062513 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.062664 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.062997 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.063096 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 15:48:03.111850 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.097399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.23.130.60:36019)
0808 15:48:03.097634 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.097645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.109522 (+ 11877us) server_negotiation.cc:407] Connection header received
0808 15:48:03.109938 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.109945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.110081 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.110174 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.111567 (+ 1393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49746: BlockingRecv error: recv got EOF from 127.0.0.1:49746 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:48:03.160215 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.145743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.23.130.61:39109)
0808 15:48:03.145935 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.145946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.152826 (+ 6880us) server_negotiation.cc:407] Connection header received
0808 15:48:03.153363 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.153370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.153489 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.153556 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.154319 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.155654 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.157797 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.158543 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.158645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.159279 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.159460 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.159525 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.159780 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.159880 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:48:03.204177 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.194926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.23.130.60:36019)
0808 15:48:03.195128 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.195135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.202107 (+ 6972us) server_negotiation.cc:407] Connection header received
0808 15:48:03.202644 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.202651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.202749 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.202825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.203904 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49762: BlockingRecv error: recv got EOF from 127.0.0.1:49762 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:48:03.259735 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.244478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.23.130.61:39109)
0808 15:48:03.244682 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.244688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.252390 (+ 7702us) server_negotiation.cc:407] Connection header received
0808 15:48:03.252928 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.252933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.253036 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.253105 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.253908 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.255028 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.257152 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.257858 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.257963 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.258728 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.258930 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.259000 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.259309 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.259424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 15:48:03.313282 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.290851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.23.130.60:36019)
0808 15:48:03.291057 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.291063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.311215 (+ 20152us) server_negotiation.cc:407] Connection header received
0808 15:48:03.311589 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.311596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.311708 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.311789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.312999 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49764: BlockingRecv error: recv got EOF from 127.0.0.1:49764 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:48:03.377408 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.363146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.23.130.61:39109)
0808 15:48:03.363336 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.363342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.370389 (+ 7047us) server_negotiation.cc:407] Connection header received
0808 15:48:03.370939 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.370946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.371048 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.371118 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.371821 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.372957 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.375076 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.375740 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.375847 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.376523 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.376688 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.376752 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.377022 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.377116 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:48:03.439599 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.407387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.23.130.60:36019)
0808 15:48:03.407633 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.407640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.437462 (+ 29822us) server_negotiation.cc:407] Connection header received
0808 15:48:03.437885 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.437893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.437999 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.438077 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.439346 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49770: BlockingRecv error: recv got EOF from 127.0.0.1:49770 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 15:48:03.461642 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.454637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.23.130.61:39109)
0808 15:48:03.454866 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.454874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.455435 (+ 561us) server_negotiation.cc:407] Connection header received
0808 15:48:03.455613 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.455618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.455720 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.455829 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.456281 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.457426 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.459444 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.460134 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.460280 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.460778 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.460941 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.461032 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.461187 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.461271 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:48:03.484512 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.476631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.23.130.60:36019)
0808 15:48:03.476830 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.476837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.482987 (+ 6150us) server_negotiation.cc:407] Connection header received
0808 15:48:03.483168 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.483174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.483274 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.483385 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.484222 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49774: BlockingRecv error: recv got EOF from 127.0.0.1:49774 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:48:03.508430 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.503607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.23.130.61:39109)
0808 15:48:03.503863 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.503869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.504139 (+ 270us) server_negotiation.cc:407] Connection header received
0808 15:48:03.504297 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.504302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.504370 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.504451 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.504829 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.505569 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.506949 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.507345 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.507439 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.507770 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.507879 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.507944 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.508073 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.508139 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 15:48:03.516085 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.514027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.23.130.60:36019)
0808 15:48:03.514247 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.514254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.514603 (+ 349us) server_negotiation.cc:407] Connection header received
0808 15:48:03.514828 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.514832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.514903 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.515015 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.515846 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49788: BlockingRecv error: recv got EOF from 127.0.0.1:49788 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:48:03.557072 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.551816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.23.130.61:39109)
0808 15:48:03.552076 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.552083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.552502 (+ 419us) server_negotiation.cc:407] Connection header received
0808 15:48:03.552668 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.552674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.552771 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.552880 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.553354 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.554146 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.555514 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.555956 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.556049 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.556459 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.556587 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.556654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.556781 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.556843 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 15:48:03.565016 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.563218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.23.130.60:36019)
0808 15:48:03.563484 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.563491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.563773 (+ 282us) server_negotiation.cc:407] Connection header received
0808 15:48:03.563889 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.563893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.563984 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.564073 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.564789 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49790: BlockingRecv error: recv got EOF from 127.0.0.1:49790 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:48:03.637980 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.632564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.23.130.61:39109)
0808 15:48:03.632834 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.632840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.633130 (+ 290us) server_negotiation.cc:407] Connection header received
0808 15:48:03.633389 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.633395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.633475 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.633587 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.634041 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.634823 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.636164 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.636630 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.636772 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.637191 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.637300 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.637431 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.637597 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.637684 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260808 15:48:03.645557 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.643680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.23.130.60:36019)
0808 15:48:03.643918 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.643926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.644244 (+ 318us) server_negotiation.cc:407] Connection header received
0808 15:48:03.644418 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.644422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.644498 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.644583 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.645263 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49798: BlockingRecv error: recv got EOF from 127.0.0.1:49798 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:48:03.782073 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:03.776874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.23.130.61:39109)
0808 15:48:03.777051 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.777056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.777380 (+ 324us) server_negotiation.cc:407] Connection header received
0808 15:48:03.777518 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.777524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.777599 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.777687 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:03.778100 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.779015 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.780331 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:03.780787 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:03.780901 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:03.781337 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:03.781441 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:03.781514 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:03.781790 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:48:03.781862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 15:48:03.790241 17125 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:03.788254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.23.130.60:36019)
0808 15:48:03.788491 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:03.788499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:03.788770 (+ 271us) server_negotiation.cc:407] Connection header received
0808 15:48:03.788981 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:03.788985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:03.789056 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:03.789162 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:03.790057 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49810: BlockingRecv error: recv got EOF from 127.0.0.1:49810 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:48:03.813776 17174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: acquiring CA information for follower catalog manager: success
I20260808 15:48:03.815657 17174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccd0a11e5a564253903b5ea3688c5c98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:48:04.055790 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:04.050210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.23.130.61:39109)
0808 15:48:04.050503 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:04.050510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:04.050863 (+ 353us) server_negotiation.cc:407] Connection header received
0808 15:48:04.051006 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:04.051011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:04.051092 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:04.051187 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:04.051626 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.052516 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.053806 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.054516 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.054622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:04.054999 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:04.055117 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:04.055190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:04.055386 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 15:48:04.055456 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:48:04.067867 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:04.061998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.23.130.60:36019)
0808 15:48:04.062201 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:04.062207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:04.062467 (+ 260us) server_negotiation.cc:407] Connection header received
0808 15:48:04.062662 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:04.062667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:04.062823 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:04.062964 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:04.063379 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.064059 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.065419 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.065938 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.066062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:04.066690 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:04.067237 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:04.067354 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:04.067456 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 15:48:04.067529 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:48:04.081993 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:04.074313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.23.130.61:39109)
0808 15:48:04.074533 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:04.074540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:04.076254 (+ 1714us) server_negotiation.cc:407] Connection header received
0808 15:48:04.076447 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:04.076454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:04.076528 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:04.076621 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:04.077126 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.077848 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.079854 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.080497 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.080620 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:04.081132 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:04.081256 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:04.081317 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:04.081593 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:48:04.081675 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:48:04.085508 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:04.074269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.23.130.62:37105)
0808 15:48:04.078814 (+ 4545us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:04.078820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:04.078838 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:04.078960 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:04.078965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:04.079163 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:04.079336 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:04.079951 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.080909 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.082952 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.083538 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.083741 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:04.084429 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:04.084883 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:04.085004 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:04.085091 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 15:48:04.085169 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4414,"thread_start_us":4303,"threads_started":1}
I20260808 15:48:04.085503 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:04.075240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.23.130.60:36019)
0808 15:48:04.075462 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:04.075466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:04.079221 (+ 3755us) server_negotiation.cc:407] Connection header received
0808 15:48:04.079458 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:04.079465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:04.079552 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:04.079622 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:04.080306 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.081339 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.083424 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:04.084061 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:04.084201 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:04.084685 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:04.084810 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:04.084907 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:04.085054 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 15:48:04.085135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:48:04.091279 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16885
I20260808 15:48:04.092283 16953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:04.204072 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16885
I20260808 15:48:04.232609 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 16965
I20260808 15:48:04.233814 17032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:04.341537 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 16965
I20260808 15:48:04.366536 24072 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2oCF2B/build/asan/bin/kudu pid 17046
I20260808 15:48:04.367745 17114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:04.480114 24072 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2oCF2B/build/asan/bin/kudu with pid 17046
2026-08-08T15:48:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5064 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10133 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (377678 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST